Raspbian Btrfs root filesystem guide (2017-06)

RaspberryPi Btrfs

Introduction

This document will guide you through the process of configuring a boot-time mountable Btrfs root filesystem on Raspbian Jessie. We'll step through configuration of a new install of Raspbian lite on a freshly written SD card and discuss at a higher level the steps needed to convert an existing install. Additionally we'll explore leveraging Btrfs snapshots to make inexpensive weak backups, send those backups remotely as differentials and demonstrate how to revert to a prior operating system state quickly and easily.

Most of this can be done over SSH once you've prepped and installed your SD card, but you'll need physical access to plug/unplug storage and diagnose any misconfigurations during boot. I've done my best to thoroughly cover any gotchas relating to boot configuration.

Scope

This is a (relatively) quick and dirty guide, if you're interested in using Btrfs I assume you're a reasonably savvy user so I've glossed over a number of things. Most "How do I …?" or "What is …?" questions should be easily Googleable and are beyond the scope of this document. I'll attempt to explain my reasons for various choices as we proceed.

I'll be using a lot of linux commands to make things easier to read, if you'd like to learn about them you can use man {command} to do so.

I've tested everything contained in this guide on a Pi 2B and a Pi 3, YMMV on other models.

What you'll need

A Pi
Internet connectivity to/from your Pi
Physical access to your Pi for some steps
An SD card that you can erase then write a fresh Raspbian image onto.
~1.5GB extra space on the SD card to keep a backup bootable ext4 partition.

Note: A 4GB SD card will work but your available space will be very limited. I'd recommend at least 8GB capacity.

Why two filesystems?

We want to keep a bootable ext4 partition available in case of damage to our btrfs filesystem. Having this partition enables you to easily modify /boot/cmdline.txt and boot an operable system with btrfs tools available to attempt recovery. Unlike booting a recovery ramdisk image you'll be able to install or update any packages you might need and restore remote backups if needed.

Strictly speaking this isn't necessary, but not everyone following along has a linux machine they can plug their SD card into and attempt repair if things go wrong.

Caveats

Btrfs does not support swap files
If you need swap you'll need to create a dedicated swap partition or locate the swap file on a non-Btrfs partition. Most of us can simply disable swap.
Do not try to be cute and set the swapfile +NOCOW _that _won't _help.

Btrfs write amplification risk
Btrfs is a copy on write (CoW) filesystem. Due to this design certain workloads and use-cases can produce large write amplification on the underlying storage media^[1].
When combined with inexpensive flash storage devices (SD cards, USB flash drives) these workloads can, and probably will, shorten the life of the storage device.

In particular I would not locate storage backing a heavily updated database or an email server working directory on Btrfs on cheap flash. These are the worst possible cases for write amplification. If you're running one of these services make another partition using a different filesystem to hold their working data.

Mitigation

If you're concerned about flash longevity but still want to use Btrfs consider using an SSD rather than an SD card or USB flash drive to hold your filesystem. Even an inexpensive SSD in a USB housing will have drastically better performance and longevity than either and will allow you to monitor device health for early warning signs before failure.

At the time of writing 16GB mSATA SSDs are available on eBay for less than the cost of a good quality 16GB Class 10 SD card from Amazon and an external mSATA to USB3 enclosure will cost you about $20. I use this particular enclosure on both of my Pis and it's suitable for 24/7 use without overheating.

Let's do this anyway

For most of us the risk of excessive write amplification is relatively low in practice and the convenience of filesystem snapshots and differential backups are good enough reasons to go ahead anyway. The sorts of workloads that generate enough write amplification to kill media don't really apply to what most of us are doing with our devices.

In my (limited) testing I haven't seen more than a 10% write overhead compared to ext4 with journaling during normal use.

If, on the other hand, you routinely kill flash storage devices with your Pi you should probably stick with ext4 and/or switch to more durable storage.

Anyway, you've been warned.

Let's get started.

New Install

Pre Boot Preparation

Write a recent Raspbian image to your SD card using your favorite software (Etcher, win32DiskImager, dd, etc.)

I'll be using 2017-04-10 Raspbian lite for this walkthrough.
https://www.raspberrypi.org/downloads/raspbian/

Modify boot partition files on your SD card before booting

We're going to stop raspbian from automatically growing the initial ext4 partition to fill all space on your SD card during first boot so we don't have to deal with shrinking the ext4 partiton later.

We'll also enable SSH as this tutorial uses a headless machine attached to the network.

add an empty file named ssh (no extension) to the root of the SD card's boot partition
edit cmdline.txt:
- remove parameter init=/usr/lib/raspi-config/init_resize.sh
- remove parameter quiet - this will help with debugging boot problems later
optional edit: config.txt
- optional: on a headless server disable sound
- optional: overclock SD card reader on rPi 3, this speeds things up noticeably when we start rsyncing a couple gigabytes around.

First boot

Initial setup

Install the SD card in the pi, plug in ethernet, then power everything on. The Pi should be booted in ~30s or so.

find the pi on your network - your DHCP server should show "raspberrypi" as a new client.
ssh in as the default user pi with passwd: raspberry
switch to root so we save keystrokes typing sudo before every command sudo -i bash
add a new user: useradd {youruser} -s /bin/bash -m -G adm,sudo
set a password: passwd {youruser}
lock the default account: passwd -l pi
revoke default account group membership: usermod -G pi,users pi

log out completely, and ssh back in as our newly added user

switch to root again sudo -i bash
grab a couple of tools we'll need: apt-get update && apt-get install -y vim tree rsync
fix our keyboard layout in case we ever have to use the console:

vim /etc/default/keyboard
- change layout from "gb" to "us"
Feel free to use nano, rather than vim, if that works for you.
setup localization, gpu memory split, wifi country:

raspi-config
- Localization options:
  - locale to en_US.UTF-8
  - timezone to America/Los Angeles
  - wifi country to US
- Advanced options:
  - GPU memory split = 16

Set your localization appropriately.

You can setup wireless now if you like.

I'm intentionally skipping hostname changes so that we can take an image of the SD card once we've finished updating the system, this way we won't have name collisions if we re-use this image on another pi later.

Manually resizing /

We need to make a bit of space before we update our system so we're going to grow the ext4 fs to ~1.5GB. This should give us enough wiggle room to go ahead and update everything and install needed tools.

We're going to pop open our SD card device in fdisk, print the partition table and locate the beginning sector of our ext4 root fs partition, delete the partition, recreate it at the same sector with our desired size then run resize2fs to grow our ext4 filesystem to fill our newly allocated space. This is non-destructive and can be done while the filesystem is online without losing data.

root@raspberrypi:~# fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/mmcblk0: 15 GiB, 16088301568 bytes, 31422464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x84fa8189

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       8192   92159   83968   41M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      92160 2534887 2442728  1.2G 83 Linux


Command (m for help):

Note the start sector of your ext4 filesystem partition, in this case it's 92160, then delete the partition.

Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Now create a larger partition in its place using the same number and starting sector as before.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (2-4, default 2):
First sector (2048-31422463, default 2048): 92160
Last sector, +sectors or +size{K,M,G,T,P} (92160-31422463, default 31422463): +1.5G

Created a new partition 2 of type 'Linux' and of size 1.5 GiB.

Double check that the partition begins on exactly the same sector as before, then write our changes to disk.

Command (m for help): p
Disk /dev/mmcblk0: 15 GiB, 16088301568 bytes, 31422464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x84fa8189

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       8192   92159   83968   41M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      92160 3213311 3121152  1.5G 83 Linux


Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

Done, let's tell the kernel that partitions on the SD card have changed and then grow the filesystem.

root@raspberrypi:~# partprobe /dev/mmcblk0

root@raspberrypi:~# resize2fs /dev/mmcblk0p2
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mmcblk0p2 is now 390144 (4k) blocks long.

Do we have free space?

root@raspberrypi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.5G  929M  456M  68% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M   12M  451M   3% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p1   41M   21M   21M  51% /boot

We do!

Verify that our PARTUUID is still there, and that it still matches the root PARTUUID in /boot/cmdline.txt:

root@raspberrypi:~# tree /dev/disk
/dev/disk
|-- by-id
|   |-- mmc-USDU1_0x41019842 -> ../../mmcblk0
|   |-- mmc-USDU1_0x41019842-part1 -> ../../mmcblk0p1
|   `-- mmc-USDU1_0x41019842-part2 -> ../../mmcblk0p2
|-- by-label
|   `-- boot -> ../../mmcblk0p1
|-- by-partuuid
|   |-- 84fa8189-01 -> ../../mmcblk0p1
|   `-- 84fa8189-02 -> ../../mmcblk0p2
|-- by-path
|   |-- platform-3f202000.sdhost -> ../../mmcblk0
|   |-- platform-3f202000.sdhost-part1 -> ../../mmcblk0p1
|   `-- platform-3f202000.sdhost-part2 -> ../../mmcblk0p2
`-- by-uuid
    |-- 70CE-EB76 -> ../../mmcblk0p1
    `-- f2100b2f-ed84-4647-b5ae-089280112716 -> ../../mmcblk0p2

5 directories, 11 files

root@raspberrypi:~# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=84fa8189-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Looking good, PARTUUID is intact and still matches so we shouldn't have problems booting.

Updates

Let's go ahead and update our system, we'll then reboot to load any kernel/firmware updates and make sure everything is working. Updating could take some time, find something to do for a few minutes after it starts.

apt-get -y dist-upgrade

Once apt-get finishes go ahead and…

Reboot!

Pop open a terminal window and start pinging the pi, on windows start>run and ping -t raspberrypi (or by ip address,) so we know when it's back up and available again.

And reboot:
root@raspberrypi:~# reboot 0

Once the Pi is back online login and verify that everything still works.

Recommended: Backup your SD card now

This would be a good time to halt the machine and take an image of your SD card. We haven't done a ton of work so far but this saves time configuring a new image and downloading updates if something breaks or you want to image another SD card in the future.

root@raspberrypi:~# reboot -H 0 and wait for the machine to shut down.

If you're using Win32DiskImager to read from the card make sure 'Read Only Allocated Partitions' is ticked to avoid reading unallocated space on the SD card and reduce image file size.

The image should be small, less than 2GB, and should only take a minute or two to create.

Machine specific configuration

Boot the Pi and make any machine specific configuration changes that you'd like now. We skipped setting a hostname earlier in case we want to use this image to setup another Pi on the same network in the future.

Creating an Initramfs

We'll need an initramfs containing the Btrfs kernel module to mount our filesystem during boot. We'll also need to make some changes to how we're referencing our root filesystem to work around a problem booting using PARTUUIDs with an initramfs on Raspbian.

Initramfs boot configuration

First, backups

Before we make any changes to boot first make backups of cmdline.txt and config.txt. If anything goes wrong you can use another machine to revert your changes and debug.

root@raspberrypi:~# cd /boot
root@raspberrypi:/boot# cp config.txt config.bak
root@raspberrypi:/boot# cp cmdline.txt cmdline.bak

Now let's tackle a couple of problems.

The first problem

Using an initramfs with the current release of Raspbian (2017-04-10) breaks booting by PARTUUID. We'll need to reference our root filesystem by UUID in cmdline.txt from now on.

Let's look at cmdline.txt, this is passed directly to the kernel at boot and tells it where to find the root filesystem among other things. We'll break parameters out to separate lines so we can see what's going on:

sed 's/ /\n/g' cmdline.txt | tee temp








dwc_otg.lpm_enable=0
console=serial0,115200
console=tty1
root=PARTUUID=84fa8189-02
rootfstype=ext4
elevator=deadline
fsck.repair=yes
rootwait

So we're booting our root ext4 filesystem using root=PARTUUID=84fa8189-02. This is going to break when we load the initramfs that we need for our kernel module. I'm not sure why it breaks, it shouldn't, but it does so we need to work around the problem.

Our workaround is pretty easy, we'll just change the root=PARTUUID= parameter in cmdline.txt to reference our root filesystem by UUID instead.

Here are our UUIDs and labels:

root@raspberrypi:~# tree /dev/disk
/dev/disk
|-- by-id
|   |-- mmc-USDU1_0x41019842 -> ../../mmcblk0
|   |-- mmc-USDU1_0x41019842-part1 -> ../../mmcblk0p1
|   `-- mmc-USDU1_0x41019842-part2 -> ../../mmcblk0p2
|-- by-label
|   `-- boot -> ../../mmcblk0p1
|-- by-partuuid
|   |-- 84fa8189-01 -> ../../mmcblk0p1
|   `-- 84fa8189-02 -> ../../mmcblk0p2
|-- by-path
|   |-- platform-3f202000.sdhost -> ../../mmcblk0
|   |-- platform-3f202000.sdhost-part1 -> ../../mmcblk0p1
|   `-- platform-3f202000.sdhost-part2 -> ../../mmcblk0p2
`-- by-uuid
    |-- 70CE-EB76 -> ../../mmcblk0p1
    `-- f2100b2f-ed84-4647-b5ae-089280112716 -> ../../mmcblk0p2

5 directories, 11 files

Our PARTUUID corresponds to /dev/mmcblk0p2 and we can see the UUID for the same partition at the bottom of the list.

We split our cmdline.txt into separate lines and dumped it into a file named temp earlier, edit this temp file to reference the partition by UUID now:








dwc_otg.lpm_enable=0
console=serial0,115200
console=tty1
root=UUID=f2100b2f-ed84-4647-b5ae-089280112716
rootfstype=ext4
elevator=deadline
fsck.repair=yes
rootwait

Verify that your UUID is correct and then put cmdline.txt back together:

paste -d" " -s temp | tee cmdline.txt

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=UUID=f2100b2f-ed84-4647-b5ae-089280112716 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Verify that your cmdline.txt is one single line with no line breaks.

Reboot to make sure everything still works. If something has gone wrong use another machine to restore your cmdline.txt from cmdline.bak, reboot and try again.

Another problem

Raspbian currently has no way to automatically update /boot/config.txt with new initrd.img file names. Every time you apply a kernel version update the install script hooks will generate a new initrd.img with a new file name.

This will break our boot sequence.

This is easy enough to solve if you remember to edit /boot/config.txt manually every time you install a kernel update or if you can pull the SD card and edit your config.txt on another machine when your next boot fails.

It's also really easy to forget to do this, so we're going to cheat and make sure that doesn't happen by holding updates to the kernel and bootloader. You'll have to manually remove these holds before updating these packages. Make yourself a note to update /boot/config.txt to reference the new initrd.img file when you update your kernel or firmware.

Log back in and switch to root: sudo -i bash

Then flag the packages for hold:

root@raspberrypi:~# apt-mark hold raspberrypi-kernel raspberrypi-bootloader
raspberrypi-kernel set on hold.
raspberrypi-bootloader set on hold.

Creating our initrd.img

Now that we've solved Raspbian's boot problems let's create and actually use our initrd.img.

Let's specify what we need to include, then create the image:

root@raspberrypi:~# echo "btrfs" >> /etc/initramfs-tools/modules

root@raspberrypi:~# cat /etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
btrfs

root@raspberrypi:~# update-initramfs -c -k `uname -r`
update-initramfs: Generating /boot/initrd.img-4.9.24-v7+

Now let's find it and configure Raspbian to use it:

root@raspberrypi:~# cd /boot
root@raspberrypi:/boot# ls -b initrd*
initrd.img-4.9.24-v7+

To load the image at boot we'll need to edit config.txt to include the following line:


initramfs initrd.img-4.9.24-v7+ followkernel

Do this now, make sure the initrd.img file name matches yours.

note: ramfsfile=initrd.img-4.9.24-v7+ syntax in config.txt should work as well, but for some reason boot hangs when I use it.

Your config.txt should now look something like this, though some of your parameters may be different:

root@raspberrypi:/boot# cat config.txt




























































# For more options and information see
# http://rpf.io/configtxtreadme
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
#disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable the lirc-rpi module
#dtoverlay=lirc-rpi

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

gpu_mem=16

initramfs initrd.img-4.9.24-v7+ followkernel

Now reboot and make sure everything still works. If boot hangs unplug the pi and use another machine to revert your config.txt and try again.

Once the Pi is back online login and verify that the btrfs module is loading at boot:

root@raspberrypi:~# dmesg | grep -i btrfs
[    5.344837] Btrfs loaded, crc32c=crc32c-generic

If dmesg doesn't show the module loading something is going wrong. Go back to the beginning of this section, recreate your initramfs and verify that you've followed every step.

Recap

At this point you should be successfully loading an initramfs containing the Btrfs module during boot and booting from your ext4 partition using root=UUID= in /boot/cmdline.txt.

Take off your fireman's hat, I think we're done putting out boot sequence fires for now.

Btrfs Preparation

Boot the Pi if it isn't already and log in, then switch to root:

sudo -i bash

Disable swap

We need to shut off swap before we go any further, locating a swap file on a copy on write filesystem is a bad idea.

root@raspberrypi:~# dphys-swapfile swapoff
root@raspberrypi:~# dphys-swapfile uninstall
root@raspberrypi:~# systemctl stop dphys-swapfile
root@raspberrypi:~# systemctl disable dphys-swapfile

Install Btrfs-tools

root@raspberrypi:~# apt-get install -y btrfs-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  liblzo2-2
The following NEW packages will be installed:
  btrfs-tools liblzo2-2
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 462 kB of archives.
After this operation, 3,521 kB of additional disk space will be used.

...

Setting up liblzo2-2:armhf (2.08-1.2) ...
Setting up btrfs-tools (3.17-1.1) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for libc-bin (2.19-18+deb8u9) ...
Processing triggers for initramfs-tools (0.120+deb8u3) ...

root@raspberrypi:~# btrfs --version
Btrfs v3.17

A note on versions

You might have noticed that our Btrfs tools are about two and a half years out of date. This is bad; Btrfs development is moving fast and there are a huge number of fixes and feature improvements that we'd like to take advantage of. We're not going to do anything about this yet as grabbing new versions requires us to make changes to our APT configuration which could have unintended consequences.

We'll solve this later and in the process demonstrate how Btrfs filesystem snapshots can be used to to save state before making important system configuration changes.

Creating the filesystem

Let's go ahead and create our new Btrfs filesystem.

root@raspberrypi:~# fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 15 GiB, 16088301568 bytes, 31422464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x84fa8189

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       8192   92159   83968   41M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      92160 3213311 3121152  1.5G 83 Linux

root@raspberrypi:~#

We'll create our new partition on 1MB boundaries (which conveniently start 1 sector after the end of the prior partition) and just assume that it's aligned "well enough" with the page and block sizes of the nand.

root@raspberrypi:~# fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/mmcblk0: 15 GiB, 16088301568 bytes, 31422464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x84fa8189

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       8192   92159   83968   41M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      92160 3213311 3121152  1.5G 83 Linux


Command (m for help): n
Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (3,4, default 3):
First sector (2048-31422463, default 2048): 3213312
Last sector, +sectors or +size{K,M,G,T,P} (3213312-31422463, default 31422463): +13G

Created a new partition 3 of type 'Linux' and of size 13 GiB.

Command (m for help): p
Disk /dev/mmcblk0: 15 GiB, 16088301568 bytes, 31422464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x84fa8189

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192    92159    83968   41M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        92160  3213311  3121152  1.5G 83 Linux
/dev/mmcblk0p3      3213312 30476287 27262976   13G 83 Linux


Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

root@raspberrypi:~# partprobe /dev/mmcblk0

root@raspberrypi:~# tree /dev/disk
/dev/disk
├── by-id
│   ├── mmc-USDU1_0x41019842 -> ../../mmcblk0
│   ├── mmc-USDU1_0x41019842-part1 -> ../../mmcblk0p1
│   ├── mmc-USDU1_0x41019842-part2 -> ../../mmcblk0p2
│   └── mmc-USDU1_0x41019842-part3 -> ../../mmcblk0p3
├── by-label
│   └── boot -> ../../mmcblk0p1
├── by-partuuid
│   ├── 84fa8189-01 -> ../../mmcblk0p1
│   ├── 84fa8189-02 -> ../../mmcblk0p2
│   └── 84fa8189-03 -> ../../mmcblk0p3
├── by-path
│   ├── platform-3f202000.sdhost -> ../../mmcblk0
│   ├── platform-3f202000.sdhost-part1 -> ../../mmcblk0p1
│   ├── platform-3f202000.sdhost-part2 -> ../../mmcblk0p2
│   └── platform-3f202000.sdhost-part3 -> ../../mmcblk0p3
└── by-uuid
    ├── 70CE-EB76 -> ../../mmcblk0p1
    └── f2100b2f-ed84-4647-b5ae-089280112716 -> ../../mmcblk0p2

5 directories, 14 files

Jackpot, time to make our filesystem.

The default node/leaf size (16KB) for btrfs should be fine for most devices, so we'll go with the defaults.

We'll also assign a label to our filesystem to make it easier to mount.

root@raspberrypi:~# mkfs.btrfs -L sd_btrfs /dev/mmcblk0p3
Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Btrfs v3.17
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (13.00GiB) ...
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
fs created label sd_btrfs on /dev/mmcblk0p3
        nodesize 16384 leafsize 16384 sectorsize 4096 size 13.00GiB

Updating fstab

Let's add our new partition to /etc/fstab so we can mount it without typing our mount options every time.

The btrfs mount option 'ssd_spread' is intended to be used with lower quality SSDs whose controllers perform better with large extents of free space, I think the naive controller in our SD card qualifies as "low quality" so we'll go with ssd_spread as an option. We'll also use noatime to avoid excessive access time metadata updates.

root@raspberrypi:~# mkdir -p /mnt/btrfs
root@raspberrypi:~# echo "LABEL=sd_btrfs /mnt/btrfs btrfs defaults,ssd_spread,noatime,noauto 0 0" >> /etc/fstab
root@raspberrypi:~# mount /mnt/btrfs

Let's have a look at what we've done:

root@raspberrypi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.5G  936M  449M  68% /
devtmpfs        481M     0  481M   0% /dev
tmpfs           486M     0  486M   0% /dev/shm
tmpfs           486M  6.5M  480M   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           486M     0  486M   0% /sys/fs/cgroup
/dev/mmcblk0p1   41M   21M   20M  52% /boot
/dev/mmcblk0p3   13G   17M   13G   1% /mnt/btrfs

root@raspberrypi:~# column -t /etc/fstab
proc                  /proc       proc   defaults                            0  0
PARTUUID=84fa8189-01  /boot       vfat   defaults                            0  2
PARTUUID=84fa8189-02  /           ext4   defaults,noatime                    0  1
LABEL=sd_btrfs        /mnt/btrfs  btrfs  defaults,ssd_spread,noatime,noauto  0  0
root@raspberrypi:~#

Perfect.

Note:
You can also assign a label to your ext4 partition using e2label /dev/mmcblk0p2 <label here> if you like. I've set my ext4 partition to sd_ext4.

Cloning / onto our new Btrfs filesystem

Creating a subvolume to hold our root filesystem

We're going to be a little bit clever and use a subvolume to make snapshot management easier in the future.

First, let's create an empty subvolume on our Btrfs filesystem and make a directory to hold future snapshots:

root@raspberrypi:~# btrfs subv create /mnt/btrfs/\@root
Create subvolume '/mnt/btrfs/@root'

root@raspberrypi:~# mkdir -p /mnt/btrfs/snapshots

root@raspberrypi:~# tree /mnt/btrfs/
/mnt/btrfs/
├── @root
└── snapshots

2 directories, 0 files

There's our subvolume. Subvolumes are automatically mounted by their parent volume so we've isolated our root into it's own subvolume which we can mount directly without exposing the rest of the filesystem tree. This way any snapshots we take in the future won't be visible unless we mount the entire Btrfs filesystem tree explicitly.

This isn't entirely necessary but it keeps things partitioned nicely and makes sure that our snapshots aren't vulnerable to accidental changes.

Cloning the filesystem

Now let's clone the filesystem, this shouldn't take too long on a fresh install with around ~1GB data.

root@raspberrypi:~# rsync -avxHW / /mnt/btrfs/\@root/

...

var/spool/
var/spool/mail -> ../mail
var/spool/cron/
var/spool/cron/crontabs/
var/spool/rsyslog/
var/tmp/

sent 896,499,356 bytes  received 601,810 bytes  6,208,312.57 bytes/sec
total size is 902,927,704  speedup is 1.01

Preparing to boot our new filesystem

Updating our target fstab

We need to update /mnt/btrfs/@root/etc/fstab to mount things properly when we boot from our new filesystem.

Let's have another look at our UUIDs and labels:

root@raspberrypi:~# tree /dev/disk
/dev/disk
├── by-id
│   ├── mmc-USDU1_0x41019842 -> ../../mmcblk0
│   ├── mmc-USDU1_0x41019842-part1 -> ../../mmcblk0p1
│   ├── mmc-USDU1_0x41019842-part2 -> ../../mmcblk0p2
│   └── mmc-USDU1_0x41019842-part3 -> ../../mmcblk0p3
├── by-label
│   ├── boot -> ../../mmcblk0p1
│   └── sd_btrfs -> ../../mmcblk0p3
├── by-partuuid
│   ├── 84fa8189-01 -> ../../mmcblk0p1
│   ├── 84fa8189-02 -> ../../mmcblk0p2
│   └── 84fa8189-03 -> ../../mmcblk0p3
├── by-path
│   ├── platform-3f202000.sdhost -> ../../mmcblk0
│   ├── platform-3f202000.sdhost-part1 -> ../../mmcblk0p1
│   ├── platform-3f202000.sdhost-part2 -> ../../mmcblk0p2
│   └── platform-3f202000.sdhost-part3 -> ../../mmcblk0p3
└── by-uuid
    ├── 70CE-EB76 -> ../../mmcblk0p1
    ├── f2100b2f-ed84-4647-b5ae-089280112716 -> ../../mmcblk0p2
    └── f4f24622-b848-494b-b89e-853bd9a0c37c -> ../../mmcblk0p3

5 directories, 16 files

Here's our current fstab on our ext4 filesystem:

root@raspberrypi:~# column -t /etc/fstab

proc                  /proc       proc   defaults                            0  0
PARTUUID=84fa8189-01  /boot       vfat   defaults                            0  2
PARTUUID=84fa8189-02  /           ext4   defaults,noatime                    0  1
LABEL=sd_btrfs        /mnt/btrfs  btrfs  defaults,ssd_spread,noatime,noauto  0  0

And here's what we want in our new fstab:

root@raspberrypi:~# column -t /mnt/btrfs/\@root/etc/fstab

proc                  /proc           proc   defaults                                  0  0
PARTUUID=84fa8189-01  /boot           vfat   defaults                                  0  2
PARTUUID=84fa8189-02  /mnt/ext4_root  ext4   defaults,noatime,noauto                   0  2
LABEL=sd_btrfs        /mnt/btrfs      btrfs  defaults,ssd_spread,noatime,noauto        0  0
LABEL=sd_btrfs        /               btrfs  defaults,ssd_spread,noatime,subvol=@root  0  0

Go ahead and edit yours to match. Pay special attention to the fsck flags at the end (the last column of numbers) as we do /not/ want anything trying to btrfs.fsck our filesystem in an emergency. Make sure your PARTUUIDs correspond to the PARTUUIDs on your system.

Note: /mnt/ext4_root is included purely for convenience and /mnt/btrfs will be useful when we start managing snapshots.

Don't forget to create the mount point for our ext4 partition:
mkdir -p /mnt/btrfs/\@root/mnt/ext4_root

Updating `/boot/cmdline.txt`

First let's make a copy of /boot/cmdline.txt so it's easy to boot from the ext4 partition in the future:

cd /boot && cp cmdline.txt cmdline.ext4

Let's look at our UUIDs again:

root@raspberrypi:/boot# tree /dev/disk
/dev/disk
├── by-id
│   ├── mmc-USDU1_0x41019842 -> ../../mmcblk0
│   ├── mmc-USDU1_0x41019842-part1 -> ../../mmcblk0p1
│   ├── mmc-USDU1_0x41019842-part2 -> ../../mmcblk0p2
│   └── mmc-USDU1_0x41019842-part3 -> ../../mmcblk0p3
├── by-label
│   ├── boot -> ../../mmcblk0p1
│   └── sd_btrfs -> ../../mmcblk0p3
├── by-partuuid
│   ├── 84fa8189-01 -> ../../mmcblk0p1
│   ├── 84fa8189-02 -> ../../mmcblk0p2
│   └── 84fa8189-03 -> ../../mmcblk0p3
├── by-path
│   ├── platform-3f202000.sdhost -> ../../mmcblk0
│   ├── platform-3f202000.sdhost-part1 -> ../../mmcblk0p1
│   ├── platform-3f202000.sdhost-part2 -> ../../mmcblk0p2
│   └── platform-3f202000.sdhost-part3 -> ../../mmcblk0p3
└── by-uuid
    ├── 70CE-EB76 -> ../../mmcblk0p1
    ├── f2100b2f-ed84-4647-b5ae-089280112716 -> ../../mmcblk0p2
    └── f4f24622-b848-494b-b89e-853bd9a0c37c -> ../../mmcblk0p3

5 directories, 16 files

Take a look at the by-uuid branch at the bottom:
/dev/mmcblk0p2 is the partition holding our ext4 filesystem
/dev/mmcblk0p3 is the partition holding our new Btrfs filesystem

Find the UUID that points to your Btrfs partition.

Now let's edit cmdline.txt to boot from our Btrfs partition and root subvolume. We need to change our root=UUID= parameter to point to our Btrfs partition, rootfstype=ext4 to rootfstype=btrfs, add rootflags=subvol=@root and remove fsck.repair=yes.

Either edit the temp file we created earlier and put it back together with paste -d" " -s temp > cmdline.txt or edit your cmdline.txt directly.

Make sure your final cmdline.txt is one single line with no line breaks.

Your new cmdline.txt should look something like this and should use your Btrfs partition's UUID:

root@raspberrypi:/boot# cat cmdline.txt

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=UUID=f4f24622-b848-494b-b89e-853bd9a0c37c rootfstype=btrfs rootflags=subvol=@root elevator=deadline rootwait

Here it is broken out to show our changes.

root@raspberrypi:/boot# sed 's/ /\n/g' cmdline.txt








dwc_otg.lpm_enable=0
console=serial0,115200
console=tty1
root=UUID=f4f24622-b848-494b-b89e-853bd9a0c37c
rootfstype=btrfs
rootflags=subvol=@root
elevator=deadline
rootwait

Reboot

Everything should be in place, go ahead and reboot: reboot 0

If something goes wrong halt the pi, pull the SD card and copy cmdline.ext4 over cmdline.txt and try again.

Log back in and switch to root: sudo -i bash

Let's verify that everything is mounted where it should be:

root@raspberrypi:~# lsblk -f
NAME        FSTYPE LABEL    UUID                                 MOUNTPOINT
mmcblk0
├─mmcblk0p2 ext4            f2100b2f-ed84-4647-b5ae-089280112716
├─mmcblk0p3 btrfs  sd_btrfs f4f24622-b848-494b-b89e-853bd9a0c37c /
└─mmcblk0p1 vfat   boot     70CE-EB76                            /boot

Success!

Let's make a copy of our cmdline.txt so we don't lose our changes if we ever have to boot from our ext4 partition in the future:

cd /boot && cp cmdline.txt cmdline.btrfs

root@raspberrypi:/boot# ls cmdline*
cmdline.bak  cmdline.btrfs  cmdline.ext4  cmdline.txt

Now we've got backup copies we can restore in case we need to boot from the ext4 partition without losing our current configuration.

Your new Btrfs filesystem

OK. We've setup a bootable system using Btrfs. Let's start doing things with it.

We'll take advantage of our old version of Btrfs-tools and use updating the package from Raspbian unstable as an opportunity to demonstrate protecting ourselves with snapshots. We'll configure APT to pull some packages from the unstable branch of Raspbian (stretch) while using a snapshot to ensure that even if we misconfigure APT and something goes horribly wrong we can simply change our boot target and roll back.

Snapshots make potentially dangerous changes like this ridiculously convenient to explore. We don't have to worry about breaking our install or manually undoing any of a dozen things we forgot to document while testing. As long as we don't crash the machine or destroy our entire filesystem we can simply rebase our root filesystem target to a known good snapshot and reboot if things go wrong.

First, a quick look at the state of our system:

root@raspberrypi:~# lsblk -f
NAME        FSTYPE LABEL    UUID                                 MOUNTPOINT
mmcblk0
├─mmcblk0p2 ext4            f2100b2f-ed84-4647-b5ae-089280112716
├─mmcblk0p3 btrfs  sd_btrfs f4f24622-b848-494b-b89e-853bd9a0c37c /
└─mmcblk0p1 vfat   boot     70CE-EB76                            /boot

root@raspberrypi:~# btrfs fi show /
Label: 'sd_btrfs'  uuid: f4f24622-b848-494b-b89e-853bd9a0c37c
        Total devices 1 FS bytes used 913.56MiB
        devid    1 size 13.00GiB used 1.30GiB path /dev/mmcblk0p3

Btrfs v3.17

root@raspberrypi:~# btrfs subv show /
/
        Name:                   @root
        uuid:                   4cdb3079-d1bf-7b48-96f9-ae21e6338715
        Parent uuid:            -
        Creation time:          2017-06-03 20:21:55
        Object ID:              258
        Generation (Gen):       168
        Gen at creation:        12
        Parent:                 5
        Top Level:              5
        Flags:                  -
        Snapshot(s):
root@raspberrypi:~#

Our first snapshot

Log back in and switch to root: sudo -i bash

Mount the full Btrfs filesystem that we added to /etc/fstab earlier and let's take a snapshot of the current system state.

root@raspberrypi:~# mount /mnt/btrfs
root@raspberrypi:~# btrfs subv snapshot / /mnt/btrfs/snapshots/\@root-pre-APT-changes
Create a snapshot of '/' in '/mnt/btrfs/snapshots/@root-pre-APT-changes'

And here's our snapshot:

root@raspberrypi:~# btrfs subv show /
/
        Name:                   @root
        uuid:                   4cdb3079-d1bf-7b48-96f9-ae21e6338715
        Parent uuid:            -
        Creation time:          2017-06-03 20:21:55
        Object ID:              258
        Generation (Gen):       169
        Gen at creation:        12
        Parent:                 5
        Top Level:              5
        Flags:                  -
        Snapshot(s):
                                snapshots/@root-pre-APT-changes

root@raspberrypi:~# tree /mnt/btrfs/snapshots/ -L 2
/mnt/btrfs/snapshots/
└── @root-pre-APT-changes
    ├── bin
    ├── boot
    ├── dev
    ├── etc
    ├── home
    ├── lib
    ├── lost+found
    ├── media
    ├── mnt
    ├── opt
    ├── proc
    ├── root
    ├── run
    ├── sbin
    ├── srv
    ├── sys
    ├── tmp
    ├── usr
    └── var

20 directories, 0 files

There we are, a safe point to restore from if we break anything.

Updating `btrfs-tools`

Now we're ready to configure APT to search the stretch repository for the packages we want.

Here's our current APT configuration:

root@raspberrypi:~# cd /etc/apt
root@raspberrypi:/etc/apt# tree
.
├── apt.conf.d
│   ├── 01autoremove
│   ├── 01autoremove-kernels
│   ├── 20listchanges
│   ├── 50raspi
│   └── 70debconf
├── listchanges.conf
├── preferences.d
├── sources.list
├── sources.list.d
│   └── raspi.list
├── trusted.gpg
└── trusted.gpg.d

4 directories, 9 files

We're going to add the stretch repo to sources.list and add a couple new files to the preferences.d/ subdirectory to tell APT how to handle packages from Raspbian unstable.

Here are our changed files:

root@raspberrypi:/etc/apt# cat sources.list





deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi

# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi

root@raspberrypi:/etc/apt# cat preferences.d/stretch-btrfs-tools.pref




Package: btrfs*
Pin: release o=raspbian, n=stretch
Pin-Priority: 999

root@raspberrypi:/etc/apt# cat preferences.d/stretch-low-priority.pref




Package: *
Pin: release o=raspbian, n=stretch
Pin-Priority: 400

We've told APT to use the stretch repo as well as jessie but we've set a priority for all stretch packages that's lower than the default of 500 for our current release. We've also specified a very high priority for Btrfs related packages from stretch. This will allow apt-get to continue tracking packages from our current release and also track updates to the Btrfs packages from stretch as they're updated. This shouldn't get us into any trouble but if it does we'll just revert to our prior snapshot.

And here's our new APT configuration:

root@raspberrypi:/etc/apt# tree
.
├── apt.conf.d
│   ├── 01autoremove
│   ├── 01autoremove-kernels
│   ├── 20listchanges
│   ├── 50raspi
│   └── 70debconf
├── listchanges.conf
├── preferences.d
│   ├── stretch-btrfs-tools.pref
│   └── stretch-low-priority.pref
├── sources.list
├── sources.list.d
│   └── raspi.list
├── trusted.gpg
└── trusted.gpg.d

4 directories, 11 files

Let's see what happens:

root@raspberrypi:/etc/apt# apt-get update
Hit http://archive.raspberrypi.org jessie InRelease
Get:1 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB]
Get:2 http://mirrordirector.raspbian.org stretch InRelease [15.0 kB]
...

Fetched 21.5 MB in 37s (579 kB/s)
Reading package lists... Done

root@raspberrypi:/etc/apt# apt-cache policy libc6
libc6:
  Installed: 2.19-18+deb8u9
  Candidate: 2.19-18+deb8u9
  Version table:
     2.24-11 0
        400 http://mirrordirector.raspbian.org/raspbian/ stretch/main armhf Packages
 *** 2.19-18+deb8u9 0
        500 http://mirrordirector.raspbian.org/raspbian/ jessie/main armhf Packages
        100 /var/lib/dpkg/status

root@raspberrypi:/etc/apt# apt-cache policy btrfs-tools
btrfs-tools:
  Installed: 3.17-1.1
  Candidate: 4.7.3-1
  Package pin: 4.7.3-1
  Version table:
     4.7.3-1 999
        400 http://mirrordirector.raspbian.org/raspbian/ stretch/main armhf Packages
 *** 3.17-1.1 999
        500 http://mirrordirector.raspbian.org/raspbian/ jessie/main armhf Packages
        100 /var/lib/dpkg/status

Perfect, asking APT about an important package (libc6) shows that we're still tracking jessie but btrfs-tools has an update candidate from stretch. Let's upgrade btrfs-tools.

root@raspberrypi:/etc/apt# apt-get upgrade btrfs-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  btrfs-progs
The following packages will be upgraded:
  btrfs-tools
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 584 kB of archives.
After this operation, 1,297 kB of additional disk space will be used.
Do you want to continue? [Y/n]

It looks like btrfs-tools has a new dependency in stretch, so we'll go ahead and install it.

Do you want to continue? [Y/n]
Get:1 http://mirrordirector.raspbian.org/raspbian/ stretch/main btrfs-tools armhf 4.7.3-1 [16.9 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ stretch/main btrfs-progs armhf 4.7.3-1 [567 kB]
Fetched 584 kB in 1s (561 kB/s)
Reading changelogs... Done
(Reading database ... 33550 files and directories currently installed.)
Preparing to unpack .../btrfs-tools_4.7.3-1_armhf.deb ...
Unpacking btrfs-tools (4.7.3-1) over (3.17-1.1) ...
Selecting previously unselected package btrfs-progs.
Preparing to unpack .../btrfs-progs_4.7.3-1_armhf.deb ...
Unpacking btrfs-progs (4.7.3-1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for initramfs-tools (0.120+deb8u3) ...
update-initramfs: /boot/initrd.img-4.9.24-v7+ has been altered.
update-initramfs: Cannot update. Override with -t option.
Processing triggers for libc-bin (2.19-18+deb8u9) ...
Setting up btrfs-progs (4.7.3-1) ...
Processing triggers for initramfs-tools (0.120+deb8u3) ...
update-initramfs: /boot/initrd.img-4.9.24-v7+ has been altered.
update-initramfs: Cannot update. Override with -t option.
Setting up btrfs-tools (4.7.3-1) ...
Processing triggers for libc-bin (2.19-18+deb8u9) ...
root@raspberrypi:/etc/apt# btrfs --version

bash: /sbin/btrfs: No such file or directory

Oops, looks like file paths have changed. Let's tell bash to rescan executable paths.

root@raspberrypi:/etc/apt# hash -r

root@raspberrypi:/etc/apt# btrfs --version
btrfs-progs v4.7.3

And there we are.

Let's run apt-get upgrade again and make sure APT isn't trying to install a pile of packages from stretch by accident.

root@raspberrypi:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Perfect.

You can check to see which version of a package APT is going to install at any time using apt-cache policy <<package>> the same way we did earlier for libc6.

Managing snapshots

Let's take a look at working with snapshots.

First we'll take a snapshot of our current state:

root@raspberrypi:~# btrfs subv snapshot / /mnt/btrfs/snapshots/\@root-`date +'%F-%T'`
Create a snapshot of '/' in '/mnt/btrfs/snapshots/@root-2017-06-05-13:41:17'

root@raspberrypi:~# btrfs subv list /
ID 258 gen 541 top level 5 path @root
ID 264 gen 426 top level 5 path snapshots/@root-pre-APT-changes
ID 268 gen 541 top level 5 path snapshots/@root-2017-06-05-13:41:17

Now we have snapshots before, and after, making our changes to APT and updating our Btrfs tools.

Rolling back to a prior snapshot

Let's pretend for a minute that something went horribly wrong, or maybe you just don't like the idea of mixing packages on your system. Rather than undoing all of our changes we'll rebase our boot mount target and reboot.

The following is a demonstration, you don't need to actually do this. There's really no risk though as we just took a snapshot of our system with our changes intact.

root@raspberrypi:~# cd /mnt/btrfs/
root@raspberrypi:/mnt/btrfs# tree -L 2
.
├── @root
│   ├── bin
│   ├── boot
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── lost+found
│   ├── media
│   ├── mnt
│   ├── opt
│   ├── proc
│   ├── root
│   ├── run
│   ├── sbin
│   ├── srv
│   ├── sys
│   ├── tmp
│   ├── usr
│   └── var
└── snapshots
    ├── @root-2017-06-05-13:41:17
    └── @root-pre-APT-changes

23 directories, 0 files

Btrfs snapshots are treated like any other subvolume and are available in the full filesystem tree so this is easy. We can either edit our /etc/fstab and /boot/cmdline.txt to point our root filesystem directly at a snapshot by setting subvol=/snapshots/@root-pre-APT-changes or we can be lazy and just move @root and replace it. I'm going to be lazy.

root@raspberrypi:/mnt/btrfs# mv \@root/ \@root-bad

root@raspberrypi:/mnt/btrfs# mv snapshots/\@root-pre-APT-changes/ \@root

root@raspberrypi:/mnt/btrfs# tree -L 2
.
├── @root
│   ├── bin
│   ├── boot
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── lost+found
│   ├── media
│   ├── mnt
│   ├── opt
│   ├── proc
│   ├── root
│   ├── run
│   ├── sbin
│   ├── srv
│   ├── sys
│   ├── tmp
│   ├── usr
│   └── var
├── @root-bad
│   ├── bin
│   ├── boot
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── lost+found
│   ├── media
│   ├── mnt
│   ├── opt
│   ├── proc
│   ├── root
│   ├── run
│   ├── sbin
│   ├── srv
│   ├── sys
│   ├── tmp
│   ├── usr
│   └── var
└── snapshots
    └── @root-2017-06-05-13:41:17

42 directories, 0 files

And let's reboot: reboot 0

And let's verify that we've rolled back.

root@raspberrypi:~# tree /etc/apt
/etc/apt
├── apt.conf.d
│   ├── 01autoremove
│   ├── 01autoremove-kernels
│   ├── 20listchanges
│   ├── 50raspi
│   └── 70debconf
├── listchanges.conf
├── preferences.d
├── sources.list
├── sources.list.d
│   └── raspi.list
├── trusted.gpg
└── trusted.gpg.d

4 directories, 9 files
root@raspberrypi:~# btrfs --version
Btrfs v3.17

Magical.

OK, I've changed my mind and I do want to keep those changes.

root@raspberrypi:~# mount /mnt/btrfs && cd /mnt/btrfs/

root@raspberrypi:/mnt/btrfs# mv \@root snapshots/\@root-pre-APT-changes

root@raspberrypi:/mnt/btrfs# mv \@root-bad/ \@root

root@raspberrypi:/mnt/btrfs# tree -L 2
.
├── @root
│   ├── bin
│   ├── boot
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── lost+found
│   ├── media
│   ├── mnt
│   ├── opt
│   ├── proc
│   ├── root
│   ├── run
│   ├── sbin
│   ├── srv
│   ├── sys
│   ├── tmp
│   ├── usr
│   └── var
└── snapshots
    ├── @root-2017-06-05-13:41:17
    └── @root-pre-APT-changes

23 directories, 0 files
root@raspberrypi:/mnt/btrfs#

reboot 0 and log in then switch to root: sudo -i bash

root@raspberrypi:~# btrfs --version
btrfs-progs v4.7.3

root@raspberrypi:~# tree /etc/apt
/etc/apt
├── apt.conf.d
│   ├── 01autoremove
│   ├── 01autoremove-kernels
│   ├── 20listchanges
│   ├── 50raspi
│   └── 70debconf
├── listchanges.conf
├── preferences.d
│   ├── stretch-btrfs-tools.pref
│   └── stretch-low-priority.pref
├── sources.list
├── sources.list.d
│   └── raspi.list
├── trusted.gpg
└── trusted.gpg.d

4 directories, 11 files

And there we are, back up and running with our updated Btrfs tools and our changes to APT intact.

Deleting snapshots

Let's make another snapshot and then delete it:

root@raspberrypi:~# mount /mnt/btrfs

root@raspberrypi:~# btrfs subv snapshot / /mnt/btrfs/temp
Create a snapshot of '/' in '/mnt/btrfs/temp'

root@raspberrypi:~# btrfs subv list /
ID 258 gen 555 top level 5 path @root
ID 264 gen 550 top level 5 path snapshots/@root-pre-APT-changes
ID 268 gen 541 top level 5 path snapshots/@root-2017-06-05-13:41:17
ID 269 gen 555 top level 5 path temp

root@raspberrypi:~# btrfs subv delete /mnt/btrfs/temp/
Delete subvolume (no-commit): '/mnt/btrfs/temp'

root@raspberrypi:~# btrfs subv list /
ID 258 gen 555 top level 5 path @root
ID 264 gen 550 top level 5 path snapshots/@root-pre-APT-changes
ID 268 gen 541 top level 5 path snapshots/@root-2017-06-05-13:41:17

Easy.

Disk usage

Snapshots are cheap, the storage cost is only the difference between the current state of the filesystem and the state at the time of snapshot creation.

Let's look at our disk usage compared to our prior ext4 filesystem:

root@raspberrypi:~# btrfs fi df /
Data, single: total=2.01GiB, used=1.09GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=264.00MiB, used=40.12MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

root@raspberrypi:~# mount /mnt/ext4_root/ && df -h /mnt/ext4_root/
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  1.5G  941M  444M  68% /mnt/ext4_root

So we're using a bit more space than we were with ext4, not much though and the bulk of that is probably stuff APT has cached since we started making changes. Keep in mind also that we're keeping two snapshots of the past state of our filesystem.

A Note on snapshots and backups

Now that we've demonstrated the convenience of snapshots I need to stress this point:

Snapshots are not backups.

Snapshots are nothing more than a feature of the Btrfs filesystem. If the filesystem becomes damaged there's no guarantee that your snapshots will be intact or even available. If you need backups, keep full backups on other media.

Backups: Full and Differential

First a note: btrfs send can only operate on read-only subvolumes so we'll need to make snapshots for this that are flagged read-only or we'll need to set the read-only flag on an existing snapshot.

Let's look at the state of our filesystem:

root@raspberrypi:/mnt/btrfs# btrfs sub show /
/
        Name:                   @root
        UUID:                   4cdb3079-d1bf-7b48-96f9-ae21e6338715
        Parent UUID:            -
        Received UUID:          -
        Creation time:          2017-06-03 20:21:55 -0700
        Subvolume ID:           258
        Generation:             839
        Gen at creation:        12
        Parent ID:              5
        Top level ID:           5
        Flags:                  -
        Snapshot(s):
                                snapshots/@root-pre-APT-changes
                                snapshots/@root-2017-06-05-13:41:17

We have two existing snapshots. The first we took before making changes to APT, the second after. Unfortunately in a prior example I mounted the pre-APT-changes snapshot and booted from it, so there are some file changes in that snapshot that differ from our current root subvolume. We'll ignore that snapshot and work with the later, post-APT-change, snapshot.

First we'll need to set our snapshot RO:

root@raspberrypi:/mnt/btrfs# btrfs prop set -ts snapshots/\@root-2017-06-05-13\:41\:17/ ro true

Now we'll take another, read-only, snapshot of our current state:

root@raspberrypi:/mnt/btrfs# btrfs sub snap -r / snapshots/\@root-backup-example
Create a readonly snapshot of '/' in 'snapshots/@root-backup-example'

For this demonstration I've plugged in a 16GB SSD with an empty ext4 filesystem and mounted it at /mnt/ssd_ext4:

root@raspberrypi:/mnt/btrfs# lsblk -f
NAME        FSTYPE LABEL    UUID                                 MOUNTPOINT
sda
└─sda1      ext4   ssd_ext4 94170e21-7a6d-4e90-b83d-70a2a4b46bc8 /mnt/ssd_ext4
mmcblk0
├─mmcblk0p2 ext4   sd_ext4  f2100b2f-ed84-4647-b5ae-089280112716
├─mmcblk0p3 btrfs  sd_btrfs f4f24622-b848-494b-b89e-853bd9a0c37c /mnt/btrfs
└─mmcblk0p1 vfat   boot     70CE-EB76                            /boot

Let's send a full backup of our first snapshot to that filesystem as a file:

root@raspberrypi:/mnt/btrfs# btrfs send -v snapshots/\@root-2017-06-05-13\:41\:17/ -f /mnt/ssd_ext4/btrfs_full_backup
At subvol snapshots/@root-2017-06-05-13:41:17/
BTRFS_IOC_SEND returned 0
joining genl thread

root@raspberrypi:/mnt/btrfs# ls -lah /mnt/ssd_ext4/
total 978M
drwxr-xr-x 3 root root 4.0K Jun  6 18:39 .
drwxr-xr-x 1 root root   44 Jun  6 17:52 ..
-rw------- 1 root root 978M Jun  6 18:40 btrfs_full_backup
drwx------ 2 root root  16K Jun  6 17:51 lost+found

Now let's look at sending a differential of our changes between that snapshot and our newly created @root-backup-example.

root@raspberrypi:/mnt/btrfs# btrfs send -v -p snapshots/\@root-2017-06-05-13\:41\:17/ snapshots/\@root-backup-example/ -f /mnt/ssd_ext4/btrfs_incremental
At subvol snapshots/@root-backup-example/
BTRFS_IOC_SEND returned 0
joining genl thread

root@raspberrypi:/mnt/btrfs# ls -lah /mnt/ssd_ext4/
total 1.2G
drwxr-xr-x 3 root root 4.0K Jun  6 20:49 .
drwxr-xr-x 1 root root   44 Jun  6 17:52 ..
-rw------- 1 root root 978M Jun  6 18:40 btrfs_full_backup
-rw------- 1 root root 196M Jun  6 20:49 btrfs_incremental
drwx------ 2 root root  16K Jun  6 17:51 lost+found

And there we are, our changes between the time we created the two snapshots total about 200MB. Note that I've done a bunch of experimenting on the demo pi in the last day, you probably won't have such a large delta.

There's something else important going on here that isn't obvious. File data and metadata are stored separately on our filesystem, so if the metadata for a file changes only this metadata change is stored in the differential backup. Contrast this with rsync which has no method to transfer just the metadata and transfers a new copy of the file if the metadata has changed. This means that btrfs send/receive can, under the right circumstances, create drastically smaller differentials than rsync would.

We can also play some neat tricks. For example, if we pipe btrfs send over ssh to a remote machine we can compress our backups remotely, rather than using the pi, on a potentially much more capable host.

A Remote Backup Comparison

Note: For the following demonstration I've upgraded the xz-utils package on our demo machine to the stretch release. The version available on jessie doesn't feature multi-threading.

apt-get upgrade xz-utils/stretch if you'd like to perform these tests yourself.

I've also installed the python program glances so I can easily monitor IO performance and CPU load in another terminal window.

Local Backup, with Compression

Let's try something interesting as a demonstration.

First we'll send a full backup of our initial timestamped snapshot locally but we'll pipe it through xz utilizing 4 threads before writing it to disk. We'll time the entire operation.

root@raspberrypi:/mnt/btrfs# time btrfs send -v snapshots/\@root-2017-06-05-13\:41\:17/ | xz -c -T 4 > /mnt/ssd_ext4/btrfs_full.xz
At subvol snapshots/@root-2017-06-05-13:41:17/

This is going to take some time.

Assuming the output of /opt/vc/bin/vcgencmd measure_temp is correct the SoC is hovering around ~75-80c during this operation. Oof, the rPi 3 isn't exactly a powerhouse for tasks like this.

root@raspberrypi:/mnt/btrfs# time btrfs send -v snapshots/\@root-2017-06-05-13\:41\:17/ | xz -c -T 4 > /mnt/ssd_ext4/btrfs_full.xz
At subvol snapshots/@root-2017-06-05-13:41:17/
BTRFS_IOC_SEND returned 0
joining genl thread

real    13m1.580s
user    47m29.010s
sys     0m50.810s

root@raspberrypi:/mnt/btrfs# ls -lah /mnt/ssd_ext4/
total 350M
drwxr-xr-x 2 root root 4.0K Jun  6 22:55 .
drwxr-xr-x 1 root root   44 Jun  6 17:52 ..
-rw-r--r-- 1 root root 350M Jun  6 22:54 btrfs_full.xz

root@raspberrypi:/mnt/btrfs# xz -l /mnt/ssd_ext4/btrfs_full.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1      41    349.1 MiB    977.6 MiB  0.357  CRC64   /mnt/ssd_ext4/btrfs_full.xz

OK, 13 minutes isn't bad for compressing a ~1GB backup on the Pi.

Remote Backup, with Compression on the remote host

Now let's have some real fun: Let's send our full backup to a remote server and compress it on the receiving end so we don't tie up resources on the Pi.

root@raspberrypi:/mnt/btrfs# time btrfs send -v snapshots/\@root-2017-06-05-13\:41\:17/ | ssh root@meta "xz -c -T 4 > /alpha/btrfs_remote_full.xz"
At subvol snapshots/@root-2017-06-05-13:41:17/
root@meta's password:

Here's the system state during the transfer:

Good, we're leaving our local resources available and temps (not shown) are around ~55C.

root@raspberrypi:/mnt/btrfs# time btrfs send -v snapshots/\@root-2017-06-05-13\:41\:17/ | ssh root@meta "xz -c -T 4 > /alpha/btrfs_remote_full.xz"
At subvol snapshots/@root-2017-06-05-13:41:17/
root@meta's password:
BTRFS_IOC_SEND returned 0
joining genl thread

real    3m54.947s
user    1m2.290s
sys     1m17.010s

root@raspberrypi:/mnt/btrfs# ssh root@meta "ls -lah /alpha/*.xz"
root@meta's password:
-rw-r--r--. 1 root root 350M Jun  6 23:03 /alpha/btrfs_remote_full.xz

Not bad. My NAS isn't much of a powerhouse either with a low clock dual core Pentium but it still outperforms the pi on tasks like this.

Let's quickly look at sending our differential in the same way.

root@raspberrypi:/mnt/btrfs# time btrfs send -v -p snapshots/\@root-2017-06-05-13\:41\:17/ snapshots/\@root-backup-example/ | ssh root@meta "xz -c -T 4 > /alpha/btrfs_remote_diff.xz"
At subvol snapshots/@root-backup-example/
root@meta's password:
BTRFS_IOC_SEND returned 0
joining genl thread

real    0m54.670s
user    0m10.850s
sys     0m11.930s

root@raspberrypi:/mnt/btrfs# ssh root@meta "ls -lah /alpha/*.xz"
root@meta's password:
-rw-r--r--. 1 root root  42M Jun  6 23:12 /alpha/btrfs_remote_diff.xz
-rw-r--r--. 1 root root 350M Jun  6 23:03 /alpha/btrfs_remote_full.xz

So there we are, secure remote differential backup with compression performed on the receiving host rather than the Pi.

Assuming a ~4GB data set size our compressed backup would take about 15 minutes to send to the remote host versus an hour to compress locally on the pi.

TODO

Existing system conversion

Let's talk a bit about converting an existing Raspbian install to Btrfs. There are too many possible scenarios for me to cover every case and write a full walkthrough but I can give you a rough overview of the steps that you'll need to follow.

First, decide whether you want to convert a portion of your SD card to a Btrfs partition.

If you're interested in trying Btrfs but don't want to commit to blowing away your existing install I suggest you temporarily use a USB flash drive to try things out. You'll need to install btrfs-tools/btrfs-progs with APT, create a partition + Btrfs filesystem on the flash drive and clone your root fs over with rsync. Next: setup your initramfs to include the btrfs module and make the relevant changes in /boot/config.txt to load the initrd.img, verify that the image loads at boot and then boot from the USB partition. You should be able to follow the general flow of the guide above and modify your partition locations accordingly. There are well-written guides available online re: mounting a USB drive partition as your root filesystem, if you're unsure about the process it would be a good idea to read one. Pay attention to the gotchas in the initramfs section above as they can make things difficult.

If you want to run Btrfs on your SD card you'll probably need to shrink your existing ext4 partition. Pull the SD card and use GParted or Paragon Partition Manager (there's a free version available) on another machine to shrink your partition. This should be non-destructive, but make a backup of the SD card first anyway. Once that's done go ahead and follow the the outline of the guide for a new install while making any needed changes for your setup. If you don't want to waste space keeping a full copy of your current install on the ext4 partition you can clone your install to a flash drive temporarily, install a fresh copy of Raspbian lite on the SD card, follow the guide above, and instead of cloning the new install to your Btrfs partition rsync your backup from the flash drive instead.

TL;DR: Read the guide for a new install so you understand what's going on, then dive in and modify it to suite your needs. Pay attention to the gotchas re: initramfs as the problems booting PARTUUIDs in Raspbian jessie with an initramfs can throw the process off.

Final notes

It's probably worth booting your ext4 recovery partition and making changes to APT to install btrfs-tools/stretch and, if you're making remote backups using xz, xz-utils/stretch to take advantage of multithreading.

Hopefully this document is useful to you, it took a lot longer to write than I planned.

tags: `raspbian` `btrfs` `raspberry pi` `linux`

See: 'Recursive Updates in Copy-on-write File Systems - Modeling and Analysis' ↩︎

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Raspbian Btrfs root filesystem guide (2017-06)

Introduction

Scope

What you'll need

Why two filesystems?

Caveats

Mitigation

Let's do this anyway

New Install

Pre Boot Preparation

Modify boot partition files on your SD card before booting

First boot

Initial setup

Manually resizing /

Updates

Reboot!

Recommended: Backup your SD card now

Machine specific configuration

Creating an Initramfs

Initramfs boot configuration

First, backups

The first problem

Another problem

Creating our initrd.img

Recap

Btrfs Preparation

Disable swap

Install Btrfs-tools

A note on versions

Creating the filesystem

Updating fstab

Cloning / onto our new Btrfs filesystem

Creating a subvolume to hold our root filesystem

Cloning the filesystem

Preparing to boot our new filesystem

Updating our target fstab

Updating /boot/cmdline.txt

Reboot

Your new Btrfs filesystem

Our first snapshot

Updating btrfs-tools

Managing snapshots

Rolling back to a prior snapshot

Deleting snapshots

Disk usage

A Note on snapshots and backups

Backups: Full and Differential

A Remote Backup Comparison

Local Backup, with Compression

Remote Backup, with Compression on the remote host

TODO

Existing system conversion

Final notes

tags: raspbian btrfs raspberry pi linux

Updating `/boot/cmdline.txt`

Updating `btrfs-tools`

tags: `raspbian` `btrfs` `raspberry pi` `linux`