# cloud-init on FreeBSD: rebooted

## Achievements So Far

- (with python netlink patches merged)
- Unittests now run successfully on FreeBSD
- bring mount(8) parsing on par with Linux' `/proc/mounts`
- make rc scripts into templates that can be relocated in ports builds
- Ephemeral Networking (DHCP and v4)
- remove `datasource_list` restrictions on BSD
- add ds-identify rc script: With `datasource_list` restrictions cut, we now need something to heavily cut down the time of guessing which cloud we're running on
- cc_rsyslog: multi-platform support
- simplify finding MBR partitions on FreeBSD and Dragonfly
- cc_ntp: Sync up with [IPv6 enabled ntp.conf](https://reviews.freebsd.org/D39954)
- Fix user account locking
- cc_growpart: add our growfs service as class for growing filesystems
- fix FreeBSD rc scripts to run in correct order
- implemented ifconfig(8) parser
- cc_ntp: Add support for BSDs
- make Makefile make agnostic
- distros/manage_services: add support to disable service
- OpenBSD: remove unneccessary pkg_cmd_environ function
- improve BSD support for distro `manage_service`
- [style: prefer absolute imports over relative imports](https://github.com/canonical/cloud-init/pull/1760) (I forgot I had done that)

<details><summary><strong>Before this project</strong></summary>

### Before this project

- growfs: [Fix UFS resizing](https://github.com/canonical/cloud-init/pull/655)
- [replace usage of dmidecode with kenv on FreeBSD](https://github.com/canonical/cloud-init/pull/621) (This makes it possible to build cloud-init cross platform)
- growfs: [split read_fs_info into linux & freebsd parts](https://github.com/canonical/cloud-init/pull/625)
- remove unnecessary reboot from gpart resize (#646)
- util: fix mounting of vfat on *BSD (#637)
- enable Puppet, Chef mcollective in default config (#385)
- make finding libc platform independent (#366)
- import sysvinit patches from freebsd-ports tree (#161)
- util: move uptime's else branch into its own boottime function (#53)
- fix unlocking method on FreeBSD (*This seems to have also **broken** locking*)
- set_passwords: support for FreeBSD (#46)
- FreeBSD: fix for get_linux_distro() and lru_cache (#59)
- distro: correctly set usr_lib_exec path for FreeBSD distro (#40)

### Before cloud-init was even in git

- resizefs: Prefix discovered devpath with '/dev/' when path does not exist

</details>

## Stumbling Stones

- cloud-init team's unclear schedules
- releases often mean two weeks without reviews
- [this fuckin thing](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273122).
- access to amphetamines and other (mental) health facilities.

## (Definite) Goals

- Network refactor
- IPv6 Support
- Handbook chapter
- CI integration via GCP/AWS/Azure
- /run migration to /var/run
- support all distro-agnostic modules
- Promote FreeBSD functionality to Cloud Providers

## (Lofty) Goals

- WiFi Support
- add support for custom pkg repos
- Implement libxo(3) for ifconfig(8) and other tools we use (parse), and where it’s missing
- virtio vsock module
- Go vsock runtime
- LXD/incus client port
- CI integration via LXD

## Methodology

- [fix all filed \*BSD bugs](https://github.com/canonical/cloud-init/issues?q=is%3Aissue+is%3Aopen+FreeBSD+OR+OpenBSD+OR+Dragonfly+OR+Dragonflybsd+OR+BSD+OR+NetBSD)
- weekly check-in, after a month we can figure out if we reduce it to bi-weekly
- interleave difficult and easy work
- interleave coding with documentation
- interleave impossible work with… uh, hand-holding?

### Modules

- ca_certs
- disk_setup
- install_hotplug: translate to devd
- keyboard
- resolv_conf
- ssh_import_id
- wireguard

### Modules to review

- mcollective
- mounts
- puppet
- chef
- seed_random
- user_groups (setting HOME)

### Networking Refactor

Migrate functions from `cloudinit.net` to `cloudinit.distros.networking`.
Many functions in `cloudinit.net` already specialized Linux/FreeBSD/OpenBSD/NetBSD implementations.
But just as many only have Linux specific code, where we error out on BSD.
The goal here is to move these functions to `cloudinit.distros.networking`, where their specialized implementations are split into distro specific classes, which are then called.

The main challenges are that

- much of the code is intertwined, so a sensible order needs to be observed.
- many cloud datasources and their helpers haven't been switched to use the distro classes
- some cloud datasources might not have a distro class at hand in a context
- Unittests need to be migrated as well, and some of these are quite crusty, and thus need modernisation

We can roughly estimate about 1 week per function, and about 1 week to get the unittests moved, and 1 week for the review and clean up.
Note that during certain times reviewers might be held up, so review cycles could sometimes be closer to 2 weeks.

#### NetworkState(Interpreter)

- find_interface_name_from_mac
- get_interfaces_by_mac

are only used by these two classes.
However, they depend on other `cloudinit.net` functions.
The NetworkState classes are used as type for settings function.
How we can migrate thse functions to the `cloudinit.distros.networking` classes and functions?

#### netinfo

netinfo duplicates a lot of `cloudinit.net` code.
This should be refactored, but is not our primary goal.

#### `cloudinit.net`

- figure out [is_renamed / driver](https://reviews.freebsd.org/D42721)
- move `get_ib_hwaddrs_by_interface()` / `get_ib_interface_hwaddr(ifname, ethernet_format)` to OpenStack

Moving order

- `_rename_interfaces()`
- `get_interfaces()`
- `interface_has_own_mac()`
- `find_candidate_nics()`
- `get_devicelist()`
- `get_interface_mac()`
- `find_interface_name_from_mac()`
- `find_interfaces_by_mac()`
- `generate_fallback_config()`
- `find_fallback_nic()`
- `is_netfail_primary()`
- `is_netfail_standby()`
- `is_netfailover()`
- `master_is_bridge_or_bond()`
- `master_is_openvswitch()`
- `get_master()`
- `device_driver()`
- `extract_physdevs()`

once these functions are refactored, we can take on all remaining `is_…()` functions:

- `is_up()`
- `is_bridge()`
- `is_bond()`
- `is_ib_interface()`
- `is_renamed()`
- `is_vlan()`

note that quite a few `cloudinit.net` functions are purely Linux specific, and don't make much sense to port, so once their dependents have been refactored, they can become private helpers:

- `get_dev_features()`
- `get_sys_class_path()`
- `sys_dev_path()`
- `read_sys_net()`
- `read_sys_net_safe()`
- `read_sys_net_int()`
- `has_netfail_standby_features()`

The remaining functions are network utility functions and can stay where they are:

- `is_disabled_cfg()`
- `has_url_connectivity()`
- `is_ip_address()`
- `is_ipv4_address()`
- `is_ipv6_address()`
- `is_ipv4_network()`
- `is_ipv6_network()`
- `natural_sort_key()`
- `is_ipv6_network()`
- `net_prefix_to_ipv4_mask()`
- `ipv4_mask_to_net_prefix()`
- `ipv6_mask_to_net_prefix()`
- `mask_and_ipv4_to_bcast_addr()`
- `subnet_is_ipv6()`

As well as the OVS functions:

- `get_ovs_internal_interfaces()`
- `is_openvswitch_internal_interface()`
- `openvswitch_is_installed()`

And the very versatile:

- `filter_hyperv_vf_with_synthetic_interface()`

#### Ephemeral classes

So far, `EphemeralDHCPv4` and `EphemeralIPv4Network` have been refactored.
We still need to add `EphemeralIPv6Network` to the mix.

### IPv6

some basic IPv6 support exists, but it's not great.
The static and SLAAC configuration need to be brought on-par with Linux.

#### DHCPv6

a bigger issue is Stateful Dynamic IPv6.
This is usually implemented with as DHCPv6 which our dhclient does not support.
There is a stalled project to [import `dhcpcd(8)`](https://reviews.freebsd.org/D22012) into base. But what we hoped would happen before the stable/14 branch has yet to happen.

## Validation

Currently, I have easy access to

- Hetzner
- Vultr (Ec2)
- AWS (Ec2)

(Although I don't find AWS particularly easy to use…)

For the this project, I would like to validate releases, which the cloud-init team issues every quarter, on as many clouds platforms as possible.

Currently, cloud-init supports the following:

- Akamai
- AliYun
- AltCloud
- Azure
- Bigstep
- CloudSigma
- CloudStack
- ConfigDrive
- DigitalOcean (deprecated in favour of ConfigDrive)
- Ec2
- Exoscale
- GCE
- Hetzner
- IBMCloud
- LXD
- MAAS
- NoCloud
- NWCS
- OpenNebula
- OpenStack
- Oracle
- OVF
- RbxCloud
- Scaleway
- SmartOS
- UpCloud
- VMware
- Vultr

of these, we definitely don't support LXD, because we don't have a virtio vsock driver, which LXD agent uses to communicate with the outside, and read cloud-init's user-data.
Further, NoCloud and ConfigDrive can be tested locally.

I would like to get help getting access to as many of these cloud platforms as possible.
Colin Percival for example me get access to AWS, by giving me $200 credits for the platform.

Once I have access to enough (or too many) of these platforms, I may have to automate my testing with Terraform.
This is a good problem to have, but might ultimately fall under "Lofty Goals".

---

# Milestone Breakdown

We tried to describe existing dependencies in the code above.

Additionally, here, we'll try to point out what key-events in the ecosystem need to happen before a milestone can even be started.
That means that as realities shift, or simply happen earlier or later than predicted, the order presented here may (or must) change as well.

External dependencies are for example the [import of `dhcpcd(8)`](https://reviews.freebsd.org/D22012), which allows for implementation of DHCPv6.

## Montly breakdown

As mentioned above, we'll assume a bug / feature / refactor takes about a week to implement. Additionally, the refactoring of Unittests will also take about a week.
With the start of the review cyclele…
We'll naïvely insert a release here every 3 months, and thus, testing of that release.

### Month 1

1. refactor `_rename_interfaces()`
2. adapt Unittests for `_rename_interfaces()`
3. review cycle / fix cc_ssh_import_id module (promised to dch@)
4. refactor `get_interfaces()`

### Month 2

1. adapt Unittests for `get_interfaces()`
2. review cycle for `get_interfaces()` / start outline for Handbook Apendix
3. refactor `interface_has_own_mac()`
4. adapt Unittests for `interface_has_own_mac()`

### Month 3

1. review cycle for `interface_has_own_mac()` / review cc_puppet module
2. refactor `get_devicelist()`
3. adapt Unittests for `get_devicelist()`
4. review cycle / test release on GCP/AWS/Azure

### Month 4

1. refactor `get_interface_mac()`
2. adapt Unittests for `get_interface_mac()`
3. review cycle / first rough draft for Handbook
4. review cycle for first draft of Handbook chapter / refactor `find_interface_name_from_mac()`

### Month 5

1. adapt Unittests for `find_interface_name_from_mac()`
2. review cycle `find_interface_name_from_mac()` / implement SLAAC for BSD
3. review cycle for SLAAC / refactor `find_interfaces_by_mac()`
4. adapt Unittests for `find_interfaces_by_mac()`

### Month 6

1. review cycle for `find_interfaces_by_mac()` / fix cc_user_groups module
2. review cycle for cc_user_groups module
3. start adding CI Integration on GCP/Azure/Aws
4. test release

### Month 7

1. refactor `generate_fallback_config()`
2. adapt Unittests for `generate_fallback_config()`
3. review cycle `generate_fallback_config()` / second draft for Handbook
4. review cycle for Handbook / fix cc_ca_certs module

### Month 8

1. review cycle for cc_ca_certs module / refactor `find_fallback_nic()`
2. adapt Unittests for `find_fallback_nic()`
3. review cycle / migrate `/run` to `/var/run` on BSD
4. review cycle for `/run` relocation

### Month 9

1. fix cc_disk_setup module
5. review cycle for cc_disk_setup module
6. refactor `is_netfail_primary()`, `is_netfail_standby()`, `is_netfailover()`
4. test release

### Month 10

1. adapt Unitests for `is_netfail_primary()`, `is_netfail_standby()`, `is_netfailover()`
2. review cycle / port cc_install_hotplug module to devd
3. review cycle for cc_install_hotplug module
4. refactor `master_is_bridge_or_bond()`, `master_is_openvswitch()`, `get_master()`

### Month 11

1. adapt Unittests for `master_is_bridge_or_bond()`, `master_is_openvswitch()`, `get_master()`
2. review cycle for `master_is_bridge_or_bond()`, `master_is_openvswitch()`, `get_master()` / fix cc_keyboard module
3. review cycle for cc_keyboard module / refactor `device_driver()`
4. adapt Unittests for `device_driver()`

### Month 12

1. review cycle for `device_driver()` / fix cc_resolv_conf module
2. review cycle for cc_resolv_conf / refactor `is_up()`, `is_bridge()`, `is_bond()`, `is_ib_interface()`, `is_renamed()`, `is_vlan()`
3. adapt Unittests for refactor `is_up()`, `is_bridge()`, `is_bond()`, `is_ib_interface()`, `is_renamed()`, `is_vlan()`
4. test release

### Month 13

1. review cycle `is_up()`, `is_bridge()`, `is_bond()`, `is_ib_interface()`, `is_renamed()`, `is_vlan()` / fix cc_wireguard module
2. review cycle for cc_wireguard module
3. Celebrate the fact that Network Refactor is OVER.
4. (Assuming [dhcpcd](https://reviews.freebsd.org/D22012) is merged) implement DHCPv6 for FreeBSD

### Month 14

1. review cycle DHCPv6 / review cc_mcollective module
2. (potential review cycle for cc_mcollective) / review cc_mounts
3. (potential review cycle for cc_mounts) / review cc_puppet
4. (potential review cycle for cc_puppet) / review cc_chef

### Month 15

1. (potential review cycle for cc_chef) / review cc_seed_random
2. create cc_pkg for configuring pkg and pkg repos
3. review cycle for cc_pkg
4. test release

## Lofty Goals

A lot of the loftier goals are left out of the above breakdown, because they may not fit into the time-constraints, and we would like to focus on delivering the best user-experience, before focusing on the developer experience.

### Out-of-Order Execution

Sometimes opportunities change, or users demand a change in priorities of features that need to be developed or bugs that need to be fixed.
Due to the granular breakup, it is quite easy to shift these things around, and to be honest, having people make such demands is probably the best thing to happen to the project, because it means more people are using, and thus testing it.