This document will guide you through the process of configuring a boot-time mountable Btrfs root filesystem on Raspbian Jessie. We'll step through configuration of a new install of Raspbian lite on a freshly written SD card and discuss at a higher level the steps needed to convert an existing install. Additionally we'll explore leveraging Btrfs snapshots to make inexpensive weak backups, send those backups remotely as differentials and demonstrate how to revert to a prior operating system state quickly and easily.
Most of this can be done over SSH once you've prepped and installed your SD card, but you'll need physical access to plug/unplug storage and diagnose any misconfigurations during boot. I've done my best to thoroughly cover any gotchas relating to boot configuration.
This is a (relatively) quick and dirty guide, if you're interested in using Btrfs I assume you're a reasonably savvy user so I've glossed over a number of things. Most "How do I …?" or "What is …?" questions should be easily Googleable and are beyond the scope of this document. I'll attempt to explain my reasons for various choices as we proceed.
I'll be using a lot of linux commands to make things easier to read, if you'd like to learn about them you can use man {command}
to do so.
I've tested everything contained in this guide on a Pi 2B and a Pi 3, YMMV on other models.
Note: A 4GB SD card will work but your available space will be very limited. I'd recommend at least 8GB capacity.
We want to keep a bootable ext4 partition available in case of damage to our btrfs filesystem. Having this partition enables you to easily modify /boot/cmdline.txt
and boot an operable system with btrfs tools available to attempt recovery. Unlike booting a recovery ramdisk image you'll be able to install or update any packages you might need and restore remote backups if needed.
Strictly speaking this isn't necessary, but not everyone following along has a linux machine they can plug their SD card into and attempt repair if things go wrong.
If you're concerned about flash longevity but still want to use Btrfs consider using an SSD rather than an SD card or USB flash drive to hold your filesystem. Even an inexpensive SSD in a USB housing will have drastically better performance and longevity than either and will allow you to monitor device health for early warning signs before failure.
At the time of writing 16GB mSATA SSDs are available on eBay for less than the cost of a good quality 16GB Class 10 SD card from Amazon and an external mSATA to USB3 enclosure will cost you about $20. I use this particular enclosure on both of my Pis and it's suitable for 24/7 use without overheating.
For most of us the risk of excessive write amplification is relatively low in practice and the convenience of filesystem snapshots and differential backups are good enough reasons to go ahead anyway. The sorts of workloads that generate enough write amplification to kill media don't really apply to what most of us are doing with our devices.
In my (limited) testing I haven't seen more than a 10% write overhead compared to ext4 with journaling during normal use.
If, on the other hand, you routinely kill flash storage devices with your Pi you should probably stick with ext4 and/or switch to more durable storage.
Anyway, you've been warned.
Let's get started.
Write a recent Raspbian image to your SD card using your favorite software (Etcher, win32DiskImager, dd, etc.)
I'll be using 2017-04-10 Raspbian lite for this walkthrough.
https://www.raspberrypi.org/downloads/raspbian/
We're going to stop raspbian from automatically growing the initial ext4 partition to fill all space on your SD card during first boot so we don't have to deal with shrinking the ext4 partiton later.
We'll also enable SSH as this tutorial uses a headless machine attached to the network.
ssh
(no extension) to the root of the SD card's boot partitioncmdline.txt
:
init=/usr/lib/raspi-config/init_resize.sh
quiet
- this will help with debugging boot problems laterconfig.txt
Install the SD card in the pi, plug in ethernet, then power everything on. The Pi should be booted in ~30s or so.
pi
with passwd: raspberry
sudo -i bash
useradd {youruser} -s /bin/bash -m -G adm,sudo
passwd {youruser}
passwd -l pi
usermod -G pi,users pi
log out completely, and ssh back in as our newly added user
switch to root again sudo -i bash
grab a couple of tools we'll need: apt-get update && apt-get install -y vim tree rsync
fix our keyboard layout in case we ever have to use the console:
vim /etc/default/keyboard
Feel free to use nano, rather than vim, if that works for you.
setup localization, gpu memory split, wifi country:
raspi-config
Set your localization appropriately.
You can setup wireless now if you like.
I'm intentionally skipping hostname changes so that we can take an image of the SD card once we've finished updating the system, this way we won't have name collisions if we re-use this image on another pi later.
We need to make a bit of space before we update our system so we're going to grow the ext4 fs to ~1.5GB. This should give us enough wiggle room to go ahead and update everything and install needed tools.
We're going to pop open our SD card device in fdisk, print the partition table and locate the beginning sector of our ext4 root fs partition, delete the partition, recreate it at the same sector with our desired size then run resize2fs to grow our ext4 filesystem to fill our newly allocated space. This is non-destructive and can be done while the filesystem is online without losing data.
Note the start sector of your ext4 filesystem partition, in this case it's 92160
, then delete the partition.
Now create a larger partition in its place using the same number and starting sector as before.
Double check that the partition begins on exactly the same sector as before, then write our changes to disk.
Done, let's tell the kernel that partitions on the SD card have changed and then grow the filesystem.
Do we have free space?
We do!
Verify that our PARTUUID is still there, and that it still matches the root PARTUUID in /boot/cmdline.txt
:
Looking good, PARTUUID is intact and still matches so we shouldn't have problems booting.
Let's go ahead and update our system, we'll then reboot to load any kernel/firmware updates and make sure everything is working. Updating could take some time, find something to do for a few minutes after it starts.
apt-get -y dist-upgrade
Once apt-get finishes go ahead and…
Pop open a terminal window and start pinging the pi, on windows start>run and ping -t raspberrypi
(or by ip address,) so we know when it's back up and available again.
And reboot:
root@raspberrypi:~# reboot 0
Once the Pi is back online login and verify that everything still works.
This would be a good time to halt the machine and take an image of your SD card. We haven't done a ton of work so far but this saves time configuring a new image and downloading updates if something breaks or you want to image another SD card in the future.
root@raspberrypi:~# reboot -H 0
and wait for the machine to shut down.
If you're using Win32DiskImager to read from the card make sure 'Read Only Allocated Partitions' is ticked to avoid reading unallocated space on the SD card and reduce image file size.
The image should be small, less than 2GB, and should only take a minute or two to create.
Boot the Pi and make any machine specific configuration changes that you'd like now. We skipped setting a hostname earlier in case we want to use this image to setup another Pi on the same network in the future.
We'll need an initramfs containing the Btrfs kernel module to mount our filesystem during boot. We'll also need to make some changes to how we're referencing our root filesystem to work around a problem booting using PARTUUIDs with an initramfs on Raspbian.
Before we make any changes to boot first make backups of cmdline.txt
and config.txt
. If anything goes wrong you can use another machine to revert your changes and debug.
Now let's tackle a couple of problems.
Using an initramfs with the current release of Raspbian (2017-04-10) breaks booting by PARTUUID. We'll need to reference our root filesystem by UUID in cmdline.txt
from now on.
Let's look at cmdline.txt
, this is passed directly to the kernel at boot and tells it where to find the root filesystem among other things. We'll break parameters out to separate lines so we can see what's going on:
sed 's/ /\n/g' cmdline.txt | tee temp
So we're booting our root ext4 filesystem using root=PARTUUID=84fa8189-02
. This is going to break when we load the initramfs that we need for our kernel module. I'm not sure why it breaks, it shouldn't, but it does so we need to work around the problem.
Our workaround is pretty easy, we'll just change the root=PARTUUID=
parameter in cmdline.txt
to reference our root filesystem by UUID instead.
Here are our UUIDs and labels:
Our PARTUUID corresponds to /dev/mmcblk0p2
and we can see the UUID for the same partition at the bottom of the list.
We split our cmdline.txt
into separate lines and dumped it into a file named temp
earlier, edit this temp file to reference the partition by UUID now:
Verify that your UUID is correct and then put cmdline.txt
back together:
paste -d" " -s temp | tee cmdline.txt
Verify that your cmdline.txt
is one single line with no line breaks.
Reboot to make sure everything still works. If something has gone wrong use another machine to restore your cmdline.txt
from cmdline.bak
, reboot and try again.
Raspbian currently has no way to automatically update /boot/config.txt
with new initrd.img file names. Every time you apply a kernel version update the install script hooks will generate a new initrd.img with a new file name.
This will break our boot sequence.
This is easy enough to solve if you remember to edit /boot/config.txt
manually every time you install a kernel update or if you can pull the SD card and edit your config.txt on another machine when your next boot fails.
It's also really easy to forget to do this, so we're going to cheat and make sure that doesn't happen by holding updates to the kernel and bootloader. You'll have to manually remove these holds before updating these packages. Make yourself a note to update /boot/config.txt
to reference the new initrd.img file when you update your kernel or firmware.
Log back in and switch to root: sudo -i bash
Then flag the packages for hold:
Now that we've solved Raspbian's boot problems let's create and actually use our initrd.img.
Let's specify what we need to include, then create the image:
Now let's find it and configure Raspbian to use it:
To load the image at boot we'll need to edit config.txt
to include the following line:
Do this now, make sure the initrd.img file name matches yours.
note: ramfsfile=initrd.img-4.9.24-v7+
syntax in config.txt
should work as well, but for some reason boot hangs when I use it.
Your config.txt should now look something like this, though some of your parameters may be different:
root@raspberrypi:/boot# cat config.txt
Now reboot and make sure everything still works. If boot hangs unplug the pi and use another machine to revert your config.txt
and try again.
Once the Pi is back online login and verify that the btrfs module is loading at boot:
If dmesg
doesn't show the module loading something is going wrong. Go back to the beginning of this section, recreate your initramfs and verify that you've followed every step.
At this point you should be successfully loading an initramfs containing the Btrfs module during boot and booting from your ext4 partition using root=UUID=
in /boot/cmdline.txt
.
Take off your fireman's hat, I think we're done putting out boot sequence fires for now.
Boot the Pi if it isn't already and log in, then switch to root:
sudo -i bash
We need to shut off swap before we go any further, locating a swap file on a copy on write filesystem is a bad idea.
You might have noticed that our Btrfs tools are about two and a half years out of date. This is bad; Btrfs development is moving fast and there are a huge number of fixes and feature improvements that we'd like to take advantage of. We're not going to do anything about this yet as grabbing new versions requires us to make changes to our APT configuration which could have unintended consequences.
We'll solve this later and in the process demonstrate how Btrfs filesystem snapshots can be used to to save state before making important system configuration changes.
Let's go ahead and create our new Btrfs filesystem.
We'll create our new partition on 1MB boundaries (which conveniently start 1 sector after the end of the prior partition) and just assume that it's aligned "well enough" with the page and block sizes of the nand.
Jackpot, time to make our filesystem.
The default node/leaf size (16KB) for btrfs should be fine for most devices, so we'll go with the defaults.
We'll also assign a label to our filesystem to make it easier to mount.
Let's add our new partition to /etc/fstab
so we can mount it without typing our mount options every time.
The btrfs mount option 'ssd_spread' is intended to be used with lower quality SSDs whose controllers perform better with large extents of free space, I think the naive controller in our SD card qualifies as "low quality" so we'll go with ssd_spread as an option. We'll also use noatime to avoid excessive access time metadata updates.
Let's have a look at what we've done:
Perfect.
Note:
You can also assign a label to your ext4 partition using e2label /dev/mmcblk0p2 <label here>
if you like. I've set my ext4 partition to sd_ext4
.
We're going to be a little bit clever and use a subvolume to make snapshot management easier in the future.
First, let's create an empty subvolume on our Btrfs filesystem and make a directory to hold future snapshots:
There's our subvolume. Subvolumes are automatically mounted by their parent volume so we've isolated our root into it's own subvolume which we can mount directly without exposing the rest of the filesystem tree. This way any snapshots we take in the future won't be visible unless we mount the entire Btrfs filesystem tree explicitly.
This isn't entirely necessary but it keeps things partitioned nicely and makes sure that our snapshots aren't vulnerable to accidental changes.
Now let's clone the filesystem, this shouldn't take too long on a fresh install with around ~1GB data.
root@raspberrypi:~# rsync -avxHW / /mnt/btrfs/\@root/
We need to update /mnt/btrfs/@root/etc/fstab
to mount things properly when we boot from our new filesystem.
Let's have another look at our UUIDs and labels:
Here's our current fstab on our ext4 filesystem:
root@raspberrypi:~# column -t /etc/fstab
And here's what we want in our new fstab:
root@raspberrypi:~# column -t /mnt/btrfs/\@root/etc/fstab
Go ahead and edit yours to match. Pay special attention to the fsck flags at the end (the last column of numbers) as we do /not/ want anything trying to btrfs.fsck our filesystem in an emergency. Make sure your PARTUUIDs correspond to the PARTUUIDs on your system.
Note: /mnt/ext4_root
is included purely for convenience and /mnt/btrfs
will be useful when we start managing snapshots.
Don't forget to create the mount point for our ext4 partition:
mkdir -p /mnt/btrfs/\@root/mnt/ext4_root
/boot/cmdline.txt
First let's make a copy of /boot/cmdline.txt
so it's easy to boot from the ext4 partition in the future:
cd /boot && cp cmdline.txt cmdline.ext4
Let's look at our UUIDs again:
Take a look at the by-uuid
branch at the bottom:
/dev/mmcblk0p2
is the partition holding our ext4 filesystem
/dev/mmcblk0p3
is the partition holding our new Btrfs filesystem
Find the UUID that points to your Btrfs partition.
Now let's edit cmdline.txt
to boot from our Btrfs partition and root subvolume. We need to change our root=UUID=
parameter to point to our Btrfs partition, rootfstype=ext4
to rootfstype=btrfs
, add rootflags=subvol=@root
and remove fsck.repair=yes
.
Either edit the temp
file we created earlier and put it back together with paste -d" " -s temp > cmdline.txt
or edit your cmdline.txt
directly.
Make sure your final cmdline.txt
is one single line with no line breaks.
Your new cmdline.txt
should look something like this and should use your Btrfs partition's UUID:
root@raspberrypi:/boot# cat cmdline.txt
Here it is broken out to show our changes.
root@raspberrypi:/boot# sed 's/ /\n/g' cmdline.txt
Everything should be in place, go ahead and reboot: reboot 0
If something goes wrong halt the pi, pull the SD card and copy cmdline.ext4
over cmdline.txt
and try again.
Log back in and switch to root: sudo -i bash
Let's verify that everything is mounted where it should be:
Success!
Let's make a copy of our cmdline.txt
so we don't lose our changes if we ever have to boot from our ext4 partition in the future:
cd /boot && cp cmdline.txt cmdline.btrfs
Now we've got backup copies we can restore in case we need to boot from the ext4 partition without losing our current configuration.
OK. We've setup a bootable system using Btrfs. Let's start doing things with it.
We'll take advantage of our old version of Btrfs-tools and use updating the package from Raspbian unstable as an opportunity to demonstrate protecting ourselves with snapshots. We'll configure APT to pull some packages from the unstable branch of Raspbian (stretch) while using a snapshot to ensure that even if we misconfigure APT and something goes horribly wrong we can simply change our boot target and roll back.
Snapshots make potentially dangerous changes like this ridiculously convenient to explore. We don't have to worry about breaking our install or manually undoing any of a dozen things we forgot to document while testing. As long as we don't crash the machine or destroy our entire filesystem we can simply rebase our root filesystem target to a known good snapshot and reboot if things go wrong.
First, a quick look at the state of our system:
Log back in and switch to root: sudo -i bash
Mount the full Btrfs filesystem that we added to /etc/fstab
earlier and let's take a snapshot of the current system state.
And here's our snapshot:
There we are, a safe point to restore from if we break anything.
btrfs-tools
Now we're ready to configure APT to search the stretch repository for the packages we want.
Here's our current APT configuration:
We're going to add the stretch repo to sources.list
and add a couple new files to the preferences.d/
subdirectory to tell APT how to handle packages from Raspbian unstable.
Here are our changed files:
root@raspberrypi:/etc/apt# cat sources.list
root@raspberrypi:/etc/apt# cat preferences.d/stretch-btrfs-tools.pref
root@raspberrypi:/etc/apt# cat preferences.d/stretch-low-priority.pref
We've told APT to use the stretch repo as well as jessie but we've set a priority for all stretch packages that's lower than the default of 500 for our current release. We've also specified a very high priority for Btrfs related packages from stretch. This will allow apt-get
to continue tracking packages from our current release and also track updates to the Btrfs packages from stretch as they're updated. This shouldn't get us into any trouble but if it does we'll just revert to our prior snapshot.
And here's our new APT configuration:
Let's see what happens:
Perfect, asking APT about an important package (libc6) shows that we're still tracking jessie but btrfs-tools
has an update candidate from stretch. Let's upgrade btrfs-tools
.
It looks like btrfs-tools
has a new dependency in stretch, so we'll go ahead and install it.
Oops, looks like file paths have changed. Let's tell bash to rescan executable paths.
And there we are.
Let's run apt-get upgrade
again and make sure APT isn't trying to install a pile of packages from stretch by accident.
Perfect.
You can check to see which version of a package APT is going to install at any time using apt-cache policy <<package>>
the same way we did earlier for libc6
.
Let's take a look at working with snapshots.
First we'll take a snapshot of our current state:
Now we have snapshots before, and after, making our changes to APT and updating our Btrfs tools.
Let's pretend for a minute that something went horribly wrong, or maybe you just don't like the idea of mixing packages on your system. Rather than undoing all of our changes we'll rebase our boot mount target and reboot.
The following is a demonstration, you don't need to actually do this. There's really no risk though as we just took a snapshot of our system with our changes intact.
Btrfs snapshots are treated like any other subvolume and are available in the full filesystem tree so this is easy. We can either edit our /etc/fstab
and /boot/cmdline.txt
to point our root filesystem directly at a snapshot by setting subvol=/snapshots/@root-pre-APT-changes
or we can be lazy and just move @root
and replace it. I'm going to be lazy.
And let's reboot: reboot 0
Login again and switch to root: sudo -i bash
And let's verify that we've rolled back.
Magical.
OK, I've changed my mind and I do want to keep those changes.
reboot 0
and log in then switch to root: sudo -i bash
And there we are, back up and running with our updated Btrfs tools and our changes to APT intact.
Let's make another snapshot and then delete it:
Easy.
Snapshots are cheap, the storage cost is only the difference between the current state of the filesystem and the state at the time of snapshot creation.
Let's look at our disk usage compared to our prior ext4 filesystem:
So we're using a bit more space than we were with ext4, not much though and the bulk of that is probably stuff APT has cached since we started making changes. Keep in mind also that we're keeping two snapshots of the past state of our filesystem.
Now that we've demonstrated the convenience of snapshots I need to stress this point:
Snapshots are nothing more than a feature of the Btrfs filesystem. If the filesystem becomes damaged there's no guarantee that your snapshots will be intact or even available. If you need backups, keep full backups on other media.
First a note: btrfs send
can only operate on read-only subvolumes so we'll need to make snapshots for this that are flagged read-only or we'll need to set the read-only flag on an existing snapshot.
Let's look at the state of our filesystem:
We have two existing snapshots. The first we took before making changes to APT, the second after. Unfortunately in a prior example I mounted the pre-APT-changes snapshot and booted from it, so there are some file changes in that snapshot that differ from our current root subvolume. We'll ignore that snapshot and work with the later, post-APT-change, snapshot.
First we'll need to set our snapshot RO:
Now we'll take another, read-only, snapshot of our current state:
For this demonstration I've plugged in a 16GB SSD with an empty ext4 filesystem and mounted it at /mnt/ssd_ext4
:
Let's send a full backup of our first snapshot to that filesystem as a file:
Now let's look at sending a differential of our changes between that snapshot and our newly created @root-backup-example
.
And there we are, our changes between the time we created the two snapshots total about 200MB. Note that I've done a bunch of experimenting on the demo pi in the last day, you probably won't have such a large delta.
There's something else important going on here that isn't obvious. File data and metadata are stored separately on our filesystem, so if the metadata for a file changes only this metadata change is stored in the differential backup. Contrast this with rsync which has no method to transfer just the metadata and transfers a new copy of the file if the metadata has changed. This means that btrfs send/receive can, under the right circumstances, create drastically smaller differentials than rsync would.
We can also play some neat tricks. For example, if we pipe btrfs send over ssh to a remote machine we can compress our backups remotely, rather than using the pi, on a potentially much more capable host.
Note: For the following demonstration I've upgraded the xz-utils
package on our demo machine to the stretch release. The version available on jessie doesn't feature multi-threading.
apt-get upgrade xz-utils/stretch
if you'd like to perform these tests yourself.
I've also installed the python program glances
so I can easily monitor IO performance and CPU load in another terminal window.
Let's try something interesting as a demonstration.
First we'll send a full backup of our initial timestamped snapshot locally but we'll pipe it through xz
utilizing 4 threads before writing it to disk. We'll time the entire operation.
This is going to take some time.
Assuming the output of /opt/vc/bin/vcgencmd measure_temp
is correct the SoC is hovering around ~75-80c during this operation. Oof, the rPi 3 isn't exactly a powerhouse for tasks like this.
OK, 13 minutes isn't bad for compressing a ~1GB backup on the Pi.
Now let's have some real fun: Let's send our full backup to a remote server and compress it on the receiving end so we don't tie up resources on the Pi.
Here's the system state during the transfer:
Good, we're leaving our local resources available and temps (not shown) are around ~55C.
Not bad. My NAS isn't much of a powerhouse either with a low clock dual core Pentium but it still outperforms the pi on tasks like this.
Let's quickly look at sending our differential in the same way.
So there we are, secure remote differential backup with compression performed on the receiving host rather than the Pi.
Assuming a ~4GB data set size our compressed backup would take about 15 minutes to send to the remote host versus an hour to compress locally on the pi.
Let's talk a bit about converting an existing Raspbian install to Btrfs. There are too many possible scenarios for me to cover every case and write a full walkthrough but I can give you a rough overview of the steps that you'll need to follow.
First, decide whether you want to convert a portion of your SD card to a Btrfs partition.
If you're interested in trying Btrfs but don't want to commit to blowing away your existing install I suggest you temporarily use a USB flash drive to try things out. You'll need to install btrfs-tools/btrfs-progs with APT, create a partition + Btrfs filesystem on the flash drive and clone your root fs over with rsync. Next: setup your initramfs to include the btrfs module and make the relevant changes in /boot/config.txt to load the initrd.img, verify that the image loads at boot and then boot from the USB partition. You should be able to follow the general flow of the guide above and modify your partition locations accordingly. There are well-written guides available online re: mounting a USB drive partition as your root filesystem, if you're unsure about the process it would be a good idea to read one. Pay attention to the gotchas in the initramfs section above as they can make things difficult.
If you want to run Btrfs on your SD card you'll probably need to shrink your existing ext4 partition. Pull the SD card and use GParted or Paragon Partition Manager (there's a free version available) on another machine to shrink your partition. This should be non-destructive, but make a backup of the SD card first anyway. Once that's done go ahead and follow the the outline of the guide for a new install while making any needed changes for your setup. If you don't want to waste space keeping a full copy of your current install on the ext4 partition you can clone your install to a flash drive temporarily, install a fresh copy of Raspbian lite on the SD card, follow the guide above, and instead of cloning the new install to your Btrfs partition rsync your backup from the flash drive instead.
TL;DR: Read the guide for a new install so you understand what's going on, then dive in and modify it to suite your needs. Pay attention to the gotchas re: initramfs as the problems booting PARTUUIDs in Raspbian jessie with an initramfs can throw the process off.
It's probably worth booting your ext4 recovery partition and making changes to APT to install btrfs-tools/stretch
and, if you're making remote backups using xz, xz-utils/stretch
to take advantage of multithreading.
Hopefully this document is useful to you, it took a lot longer to write than I planned.
raspbian
btrfs
raspberry pi
linux