Azure network metadata
====

# IMDS network metadata example cases

## single nic with public and private IP

{
   "interface" : [
      {
         "macAddress" : "000D3A047598",
         "ipv6" : {
            "ipAddress" : []
         },
         "ipv4" : {
            "subnet" : [
               {
                  "prefix" : "24",
                  "address" : "10.0.0.0"
               }
            ],
            "ipAddress" : [
               {
                  "privateIpAddress" : "10.0.0.4",
                  "publicIpAddress" : "104.46.124.81"
               }
            ]
         }
      }
   ]
}

## dual nic: 1 pub/private 2: private only

{
   "interface" : [
      {
         "ipv4" : {
            "ipAddress" : [
               {
                  "privateIpAddress" : "10.0.0.4",
                  "publicIpAddress" : "104.46.120.4"
               }
            ],
            "subnet" : [
               {
                  "address" : "10.0.0.0",
                  "prefix" : "24"
               }
            ]
         },
         "ipv6" : {
            "ipAddress" : []
         },
         "macAddress" : "000D3A047598"
      },
      {
         "ipv4" : {
            "subnet" : [
               {
                  "address" : "10.0.0.0",
                  "prefix" : "24"
               }
            ],
            "ipAddress" : [
               {
                  "privateIpAddress" : "10.0.0.6",
                  "publicIpAddress" : ""
               }
            ]
         },
         "ipv6" : {
            "ipAddress" : []
         },
         "macAddress" : "000D3A03C5AB"
      }
   ]
}

## dual nic public and private IPs (no restart needed)

{
   "interface" : [
      {
         "ipv4" : {
            "ipAddress" : [
               {
                  "privateIpAddress" : "10.0.0.4"
                  "publicIpAddress" : "104.46.120.4",
               }
            ],
            "subnet" : [
               {
                  "address" : "10.0.0.0"
                  "prefix" : "24",
               }
            ],
         },
         "ipv6" : {
            "ipAddress" : []
         },
         "macAddress" : "000D3A047598"
      },
      {
         "ipv4" : {
            "subnet" : [
               {
                  "address" : "10.0.0.0",
                  "prefix" : "24"
               }
            ],
            "ipAddress" : [
               {
                  "privateIpAddress" : "10.0.0.6",
                  "publicIpAddress" : "104.46.126.112"
               }
            ]
         },
         "ipv6" : {
            "ipAddress" : []
         },
         "macAddress" : "000D3A03C5AB"
      }
   ]
}

## Dual NIC: nic1: public/private + additional private-only nic2: private-only

{
   "network" : {
      "interface" : [
         {
            "ipv4" : {
               "ipAddress" : [
                  {
                     "publicIpAddress" : "104.46.120.4",
                     "privateIpAddress" : "10.0.0.4"
                  },
                  {
                     "publicIpAddress" : "",
                     "privateIpAddress" : "10.0.0.7"
                  }
               ],
               "subnet" : [
                  {
                     "prefix" : "24",
                     "address" : "10.0.0.0"
                  }
               ]
            },
            "macAddress" : "000D3A047598",
            "ipv6" : {
               "ipAddress" : []
            }
         },
         {
            "ipv4" : {
               "ipAddress" : [
                  {
                     "publicIpAddress" : "",
                     "privateIpAddress" : "10.0.0.6"
                  }
               ],
               "subnet" : [
                  {
                     "address" : "10.0.0.0",
                     "prefix" : "24"
                  }
               ]
            },
            "macAddress" : "000D3A03C5AB",
            "ipv6" : {
               "ipAddress" : []
            }
         }
      ]
   },
   "compute" : {
      "zone" : "",
      "publisher" : "Canonical",
      "vmScaleSetName" : "",
      "placementGroupId" : "",
      "subscriptionId" : "12aad61c-6de4-4e53-a6c6-5aff52a83777",
      "offer" : "UbuntuServer",
      "osType" : "Linux",
      "name" : "my-b1",
      "sku" : "18.04-LTS",
      "resourceGroupName" : "srugroup1",
      "vmSize" : "Standard_DS1_v2",
      "vmId" : "7dcfe5f6-f2c3-4f14-a3a2-a7444c62e156",
      "location" : "eastus2",
      "tags" : "",
      "version" : "18.04.201805220",
      "platformUpdateDomain" : "0",
      "platformFaultDomain" : "0"
   }
}

## 2 nics: nic1: priv/pub + priv nic2: priv dhcp + priv static (10.0.2.201)
* no way to tell static config from dynamic in IMDS

{
   "network" : {
      "interface" : [
         {
            "ipv4" : {
               "subnet" : [
                  {
                     "prefix" : "24",
                     "address" : "10.0.0.0"
                  }
               ],
               "ipAddress" : [
                  {
                     "privateIpAddress" : "10.0.0.4",
                     "publicIpAddress" : "104.46.120.4"
                  },
                  {
                     "publicIpAddress" : "",
                     "privateIpAddress" : "10.0.0.7"
                  }
               ]
            },
            "ipv6" : {
               "ipAddress" : []
            },
            "macAddress" : "000D3A047598"
         },
         {
            "ipv6" : {
               "ipAddress" : []
            },
            "macAddress" : "000D3A03C5AB",
            "ipv4" : {
               "subnet" : [
                  {
                     "address" : "10.0.0.0",
                     "prefix" : "24"
                  }
               ],
               "ipAddress" : [
                  {
                     "privateIpAddress" : "10.0.0.6",
                     "publicIpAddress" : ""
                  },
                  {
                     "publicIpAddress" : "",
                     "privateIpAddress" : "10.0.0.201"
                  }
               ]
            }
         }
      ]
   },
   "compute" : {
      "subscriptionId" : "12aad61c-6de4-4e53-a6c6-5aff52a83777",
      "offer" : "UbuntuServer",
      "resourceGroupName" : "srugroup1",
      "vmScaleSetName" : "",
      "tags" : "",
      "osType" : "Linux",
      "sku" : "18.04-LTS",
      "platformUpdateDomain" : "0",
      "name" : "my-b1",
      "publisher" : "Canonical",
      "version" : "18.04.201805220",
      "zone" : "",
      "placementGroupId" : "",
      "location" : "eastus2",
      "platformFaultDomain" : "0",
      "vmSize" : "Standard_DS1_v2",
      "vmId" : "7dcfe5f6-f2c3-4f14-a3a2-a7444c62e156"
   }
}

# 1 nic dual stack ipv4/ipv6

    "compute": {
     "location": "eastus2",
     "name": "test-sru-eoan-advanced",
     "offer": "UbuntuServer",
     "osType": "Linux",
     "placementGroupId": "",
     "platformFaultDomain": "0",
     "platformUpdateDomain": "0",
     "publisher": "Canonical",
     "resourceGroupName": "srugroupIPV6",
     "sku": "19.10-DAILY",
     "subscriptionId": "12aad61c-6de4-4e53-a6c6-5aff52a83777",
     "tags": "",
     "version": "19.10.201910240",
     "vmId": "ca25da9d-40d6-4772-b9ec-211c8eeebe0b",
     "vmScaleSetName": "",
     "vmSize": "Standard_DS2_v2",
     "zone": ""
    },
    "network": {
     "interface": [
      {
       "ipv4": {
        "ipAddress": [
         {
          "privateIpAddress": "192.168.0.4",
          "publicIpAddress": "137.116.62.187"
         }
        ],
        "subnet": [
         {
          "address": "192.168.0.0",
          "prefix": "24"
         }
        ]
       },
       "ipv6": {
        "ipAddress": [
         {
          "privateIpAddress": "ace:cab:deca:deed::4"
         }
        ]
       },
       "macAddress": "000D3A0363B7"
      }
     ]
    }

# Azure upgrade path cloud-init behavior
Cloud-init in tip of master now has support for regenerating network config every boot on Azure. It also removes existing CPC image support scripts or netplan config which enabled dhcp on every interface > eth0 via udev event handling (in Xenial) or netplan for >= Artful.
We need to decide what permanent behavior we have as I'm certain we have gaps for some customers if we decide to retain original network configuration for instances(pets) which upgrade cloud-init.

Here are some scenarios for which we need to decide behavior:

* **new installs cosmic+**: new instances will generate config from IMDS on every boot if there is a net config delta.
* **new installs <= bionic**: new instances will generate fallback config
* Allow for sysconfig in /etc/cloud/cloud.cfg.d:

        datasource:
          Azure:
            updates:
               policy-version: 1
               network:
                 when: [BOOT, BOOT_NEW_INSTANCE]
                 
        # this would also work here.
        updates:
           policy-version: 1
           network:
             when: [BOOT, BOOT_NEW_INSTANCE]

* *GAP: if we remove cpc udev scripts on postinst this breaks eth1+ on supported cloud-images with xenial, artful, bionic*
* **upgraded instances cosmic**: devel release; generate even if fallback was originally created
* **upgraded instances A,B**: emit a /var/lib/cloud/upgraded by preinst as marker file. leave original/cached fallback config in place
* **upgraded instances X**: leave original/cached fallback config in place
* *GAP: if we remove cpc udev scripts on postinst this breaks eth1+ on supported cloud-images with xenial, artful, bionic*
* **upgraded instances cosmic**: devel release; generate even if fallback was originally created

## new install
Frest