netplan configured custom route table
=====================================

Questions
---------
* Can networkd allow the dhcp address to be used in PreferredAddress declarations on a route?
* How should cloud-init emit netplan to setup custom routes which preserve dhcp source IP on interfaces with additional static addresses assigned? (broken example below)
* How does one create the following custom ip table rule in netplan and systemd:
* 32764: from all lookup 99
* the corresponding ip command for is: **ip rule add to any table 99**

Background
----------
In [bug 1815245](https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1815254) We are facing an issue on Azure with loss of network connectivity on Azure when adding multipe ip addresses to an existing dhcp interface on a cloud platform that restricts out-bound network traffic from non-source IPs.

Example use case
-----------------
DHCP in this example allocates the primary nic with **10.0.0.5 255.255.255.0** routed through default gateway **10.0.0.1**. The platform routers block any traffic not coming from dhcp aloocated source ip of **10.0.0.5**. So we need custom route table to ensure traffic out of the box comes from the dhcp-allocated IP address.

The following ip commands setup the custom route table which allows me to retain connectivity to the metadata service and external ip addresses once netplan/systemd adds the additional static ip **10.0.0.7** to **eth0**:

ip route add 169.254.169.254/32 dev eth0 src 10.0.0.5
ip rule add from 10.0.0.0/24 table 99
ip route add table 99 default via 10.0.0.1 src 10.0.0.5
ip rule add to any table 99

The route and rules tables look like the following after these commands on a dhcp'd eth0 vm:

ubuntu@my-d2:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:3a:7a:d6:32 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.5/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:3aff:fe7b:3b51/64 scope link 
       valid_lft forever preferred_lft forever
ubuntu@my-d2:~$ ip route show
default via 10.0.0.1 dev eth0 proto dhcp metric 100 
    10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.5 
168.63.129.16 via 10.0.0.1 dev eth0 proto dhcp metric 100 
169.254.169.254 dev eth0 scope link src 10.0.0.5 
169.254.169.254 via 10.0.0.1 dev eth0 proto dhcp metric 100 
ubuntu@my-d2:~$ ip route show table 99
default via 10.0.0.1 dev eth0 src 10.0.0.5 
ubuntu@my-d2:~$ ip rule show
0:	from all lookup local 
32764:	from all lookup 99 
32765:	from 10.0.0.0/24 lookup 99 
32766:	from all lookup main 
32767:	from all lookup default 

On Ubuntu Eoan: netplan 0.97-0ubuntu1/systemd 240-6ubuntu9
The following netplan.yaml tries to express the same config:

network:
    ethernets:
        eth0:
            addresses:
              - 10.0.0.7/24
            dhcp4: true
            match:
                macaddress: 00:0d:3a:7a:d6:32
            set-name: eth0
            routing-policy:
              - to: 10.0.0.0/24
                table: 99
              - from: 0.0.0.0/0
                table: 99
            routes:
             - to: 0.0.0.0/0
               via: 10.0.0.1
               from: 10.0.0.5
               table: 99
             - to: 168.254.169.254/32
               via: 10.0.0.1
               from: 10.0.0.5
               table: 99
             - from: 10.0.0.5
               to: 10.0.0.0/24
               via: 10.0.0.1
               table: 99
    version: 2

This results in the following /run/systemd/network/10-netplan-eth0.network

[Match]
MACAddress=00:0d:3a:7a:d6:32
Name=eth0

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6
Address=10.0.0.7/24

[Route]
Destination=0.0.0.0/0
Gateway=10.0.0.1
PreferredSource=10.0.0.5
Table=99

[Route]
Destination=168.254.169.254/32
Gateway=10.0.0.1
PreferredSource=10.0.0.5
Table=99

[Route]
Destination=10.0.0.0/24
Gateway=10.0.0.1
PreferredSource=10.0.0.5
Table=99

[RoutingPolicyRule]
To=10.0.0.0/24
Table=99

[RoutingPolicyRule]
From=0.0.0.0/0
Table=99

[DHCP]
RouteMetric=100
UseMTU=true

networkd fails to setup routes for the dhcp IP address 10.0.0.5 claiming:
**eth0: Could not set route: Invalid argument**
In IRC ddstreet lead us to the fact that dhcp addresses may not be referenced in PreferredSource values because dhcp may not have come back yet at the time the route is being configured.

If we instead specify the expected dhcp ip address in netplan's static addressees, networkd doesn't complain about the PreferredSource lines, but ip route shows static ip address as the proto type

addresses:
 - 10.0.0.5/32
 - 10.0.0.7/32

Routes now show proper src ip, but still ``proto static`` instead of ``proto dhcp``

ubuntu@my-e1:~$ ip route show table 99
default via 10.0.0.1 dev eth0 proto static src 10.0.0.5 
10.0.0.0/24 via 10.0.0.1 dev eth0 proto static src 10.0.0.5 
168.254.169.254 via 10.0.0.1 dev eth0 proto static src 10.0.0.5 

Also missing is a rule
32764: from all lookup 99

ubuntu@my-e1:~$ ip rule show
0:	from all lookup local 
0:	from all to 10.0.0.0/24 lookup 99 
32766:	from all lookup main 
32767:	from all lookup default 

Question: how do we get netplan/systemd to emit that from all lookup 99 for our custom route table?