# Fake README.md for new disconnected mirroring

## Preface

When OpenShift clusters are deployed, they require a variety of containerized software to be available in images. OpenShift clusters with connections to the internet download these images automatically as part of the installation process on day 1 or as part of consuming additional software on day 2.

For disconnected clusters without an direct internet connection these images need to be downloaded beforehand and made available for installation at day 1 and OpenShift usage on day 2 in a registry accessible to the OpenShift cluster.

You use `oc adm mirror` to prepare and maintain this offline mirror.

## Quick start:

If you are looking to jump start your disconnected deployment, use the following steps to create a minimal mirror on a standard RHEL 8 host and deploy a cluster from it:

Download binaries:

$ wget https://mirror.openshift.com/[…]/openshift-client.tar.gz
$ wget https://mirror.openshift.com/[…]/quay-install.tar.gz
$ tar -xzf openshift-client.tar.gz
$ tar -xzf quay-install.tar.gz
$ chmod u+x oc quay-install

Prepare a registry:

$ ./quay-install --all-in-one

Mirror the core release images to the registry:

$ ./oc adm mirror --operatorhub=false --samples=false --update-graph=false

Install your cluster from the registry:

$ ./openshift-install create cluster --mirror-config=./offline-manifests/

## Selecting which content to mirror

#### Content Types

Disconnected OpenShift clusters can consume the following content that would need to be mirrored:

- OpenShift Release payload (required for cluster installation and updates)
- OpenShift Update Graph (required to update the cluster)
- OpenShift OperatorHub (required for using OpenShift Add-Ons and 3rd party Operators)
- Sample Images (required for sample `ImageStreams` used by OpenShift Templates)

#### Content structure

Some of this content is offered in the context of OpenShift releases:

- OpenShift Release Payload - one per OpenShift z-stream, e.g. OpenShift 4.8.1
- OpenShift OperatorHub - multiple catalogs per OpenShift minor release, e.g. `redhat-operators:v4.8` or `certified-operators:v.4.7`

The rest is independent of a particular OpenShift release and generally published for OpenShift 4:

- OpenShift Update Graph - one image containing update information for *all* released OpenShift 4 versions
- Sample Images - images for a particular OpenShift `ImageStream`, e.g. `registry.redhat.io/rhscl/ruby-25-rhel7:latest`

#### Listing all available content

Run the following commands on an internet-connected system to discover the available downloads for the above mentioned content. `oc` will highlight the version that corresponds to its own release.

List all available OpenShift Release Payloads:

$ oc adm mirror list --releases

Listing stable channels. Append --channel=<channel-name> to filter.
Use oc adm mirror list --channels to discover other channels. 

Available OpenShift Releases:

Channel: stable-4.6
* 4.6.1
* 4.6.2
* 4.6.3
* 4.6.4
...
* 4.6.13

Channel: stable-4.7
* 4.7.0
* 4.7.1
...

Channel: stable-4.8
* 4.8.0 
* 4.8.2 (This release)
* 4.8.3
* 4.8.5

List all available OpenShift Release channels:

$ oc adm mirror list --channels

stable-4.4
fast-4.4
candidate-4.4

stable-4.5
fast-4.5
candidate-4.5

stable-4.6
fast-4.6
candidate-4.6
eus-4.6

stable-4.7
fast-4.7
candidate-4.7

stable-4.8
fast-4.8
candidate-4.8

List all available OpenShift OperatorHub catalogs:

$ oc adm mirror list --operatorhub --catalogs

Available OpenShift OperatorHub catalogs:

OpenShift 4.6:

* redhat-operators:v4.6      (60 Operators, 220GB)
* certified-operators:4.6    (102 Operators, 160GB)
* community-operators:v4.6   (180 Operators, 85GB)


OpenShift 4.7:

* redhat-operators:v4.7      (61 Operators, 222GB)
* certified-operators:4.7    (108 Operators, 162GB)
* community-operators:v4.7   (182 Operators, 86GB)

OpenShift 4.8 (This release):

* redhat-operators:v4.8      (65 Operators, 229GB)
* certified-operators:4.8    (110 Operators, 165GB)
* community-operators:v4.8   (186 Operators, 90GB)

List all available Operators in a OpenShift OperatorHub catalog:

$ oc adm mirror list --operatorhub --operators --catalog=redhat-operators:v4.8

Available Operators in redhat-operators:v4.8 (This release):

  [package name]                     Display Name

- [openshift-service-mesh]           Red Hat OpenShift Service Mesh (5 versions)
- [rhacm]                            Red Hat Advanced Cluster Management (21 versions)
- [rhacs]                            Red Hat Advanced Cluster Security (1 version)
- [openshift-container-storage]      Red Hat OpenShift Container Storage (10 versions)
- [container-native-virtualization]  Red Hat OpenShift Virtualization
...

List all available Sample Images:

$ oc adm mirror list --samples

Available sample images in ImageStreams:

  [imagestream]  Image URL

- [ruby]         registry.redhat.io/rhscl/ruby-25-rhel7:latest
- [ruby]         registry.redhat.io/rhscl/ruby-29-rhel7:latest
- [ruby]         registry.redhat.io/rhscl/ruby-35-rhel7:latest
- [python]       registry.redhat.io/rhscl/python-29-rhel7:latest
- [python]       registry.redhat.io/rhscl/python-35-rhel7:latest
- [nginx]        registry.redhat.io/rhscl/nginx-113-rhel7:latest
- [nginx]        registry.redhat.io/rhscl/nginx-116-rhel7:latest
...

## Registry Requirements

For the above content to be accessible by disconnected OpenShift cluster a registry is required with the following pre-requisites:
- be docker v2-2 compatible and OCI compliant
- sufficient storage space available (minimum 5GB, 400GB for a complete disconnected mirror of *all* content (not likely to be required))
- accessible by the host running the `openshift-install` and `oc` commands
- accessible by the OpenShift cluster nodes

## Possible scenarios

There are two supported scenarios to create a disconnected mirror:

1) There is a dual-homed bastion host that has connectivity to both, the internet and OpenShift clusters - from this host the mirror is created and subsequently the installation is orchestrated. It could also run the registry that holds the disconnected mirror.

2) There is a network air-gap between systems that have internet access and systems that run OpenShift. The mirroring needs to be carried out on an internet connected system that saves the mirror content on disk. From there the content is transported via offline media to disconnected bastion host from which the installaiton is orchestrated. This host could also run the registry that holds the disconnected mirror.

## Performing a initial disconnected mirror on a dual-homed bastion

### 1. Prepare a target registry

If you already have a registry that complies to the **#Requirements** you can skip the below step. Make sure your registry has the following namespaces already:

- `openshift`
- `operatorhub`
- `samples`

`oc adm mirror` will choose these namespaces to mirror the different content types.

If you don't have a registry yet you can easily deploy a single node Red Hat Quay registry with the following sequence of command on your dual-homed bastion host:

$ wget https://mirror.openshift.com/[…]/quay-install
$ chmod u+x quay-install
$ ./quay-install --all-in-one
...

Red Hat Quay Deployment finished!
Your registry is available at https://bastion.company.local
You can log in to the console with superuser "quayadmin" 
and password "8XEFT-4KQIh-MmCmQ-e3Dq4"

Pre-created the following organizations:
- openshift
- operatorhub
- samples

Presence of `oc` detected, automatically logged in as "superuser".

The above command creates a local Red Hat Quay deployment with the following properties:

- OCI compliant / docker v2-2 compatible registry
- uses TLS for traffic encryption
- uses the fully-qualified hostname
- automaticaly starts at boot of this system via `systemd`
- self-signed TLS certs are trusted by this system
- pre-created registry namespaces (*Organizations* in Quay) to hold mirror data

Some of this is customizable. Refer to `quay-install --all-in-one --help` for options.

This registry deployment is supported as part of your OpenShift subscription but limited to the use as registry for Red Hat OpenShift offline content.

### 2a. Create the mirror (simple selection)

On your dual-homed bastion host you use `oc adm mirror` to create the mirror. By default a registry is expected to be running on the same node on port 443. If you did not use `quay-install` on this node or you have a registry running on another node use `oc adm mirror login --registry=<fqdn>:<port>` to authenticate. Also append `--registry=<fqdn>:<port>` to all of the following commands.

You can use a set of CLI switches to select which data to mirror:

#### Mirror everything (will take a long time):

$ oc adm mirror

This command will mirror all content of the OCP release that the `oc` version corresponds to. This will take **significant** amount of time and disk space. Only do this when you are sure you want a full mirror of everything.

#### Mirror everything associated to a particular OCP release (will take a long time):

$ oc adm mirror --release 4.7

Like the previous command, but mirrors content of a particular OCP release.
You can append this to all of the commands below to select a particular OCP release instead of the one associated with the `oc` version

#### Mirror only the release payload (required minimum to install disconnected)

$ oc adm mirror --operatorhub=false --samples=false --update-graph=false

#### Mirror only all OperatorHub content (will take a long time)

$ oc adm mirror --core=false --operatorhub=true --samples=false --update-graph=false

### 2b. Create the mirror (advanced selection)

As outlined in the previous section, mirroring everything, especially the OperatorHub content can take extensive amount of time.
To further trim the selection of content that should be mirrored a configuration file can be used instead of the command line switches shown above to allow more granulary selecting content.

Create a mirror config file, that follows the below example called `mirror.yaml`:

apiVersion: v1
mirror:
 ocp:   
   channels: <1>
     - stable-4.7
   versions: <2>
     - 4.6.13
     - 4.6.15
   graph: true <3>
 operators: <4>
   - catalog: redhat-operators:v4.7 <5>
   - catalog: certified-operators:v4.7 <65>
     packages: 
       - name: couchbase-operator <7>
       - name: mongodb-operator
         version: '>1.4.0' <8>
       - name: crunchy-postgresql-operator
         channel: 'stable' <9>
   - catalog: community-operators:v4.7 <10>
     latestOnly: true
 samples: <11>
   - ruby
   - python
   - nginx

<1> A list which selects all OCP versions of a particular channels for mirroring - use `oc adm mirror list --channels` for a full listing

<2> A list which selects the desired OCP core version images - use `oc adm mirror list --releases` for a full listing

<3> A toggle whether or not to mirror the update grah image for OpenShift update service

<4> A configuration section to mirror subsets of Operator catalogs - use `oc adm mirror list --operatorhub --catalogs` for a full listing

<5> Mirror an entire catalog of a particular OCP version

<6> Mirror a subset of a catalog of a particular OCP version - use `oc adm mirror list --operatorhub --operators --catalog=redhat-operators:v4.7` for a full listing

<7> Mirror all versions of a particular Operator

<8> Mirror a particular Operator newer than a certain version (supported constraints: `<`, `<=`, `==`, `>`, `>=`)

<9> Mirror all versions of an Operator in a particular channel

<10> Mirror only the latest version of all Operator of a catalog

<11> Mirror a list of sample images referenced by `ImageStreams` - use `oc adm mirror list --samples` for a full listing

Start the mirroring process with the config file:

$ oc adm mirror -f mirror.yaml

The `-f` switch instructs `oc adm mirror` to use the referenced file to determine what content to mirror. It is mutually exclusive to all other switches.

### 3. Start the disconnected installation

At this point `oc adm mirror` has mirrored the requested content and also a set of files in a subdirectory called `offline-manifests/` which has the following contents

- `mirror-registry.json` (credentials for the `openshift-installer` about the mirror registry)
- `imagecontentsourcepolicies.yaml` (instructs running OpenShift cluster to obtain images from the mirror registry instead of registry.redhat.io)
- `samples-patch.yaml` (instructs the OpenShift Samples Operator to obtain images from the mirror registry instead of registry.redhat.io)
- `update-graph.yaml` (instructiosn OpenShift Update Service to retrieve the update graph from the mirror registry)

You instruct `openshift-install` to refer to the offline mirror by pointing to this directory:

`$ openshift-install create cluster --mirror-config=./offline-manifests/`

This will cause the installer to:

* read the `mirror-registry.json` to reach the registry that holds the core release images used during install
* apply `imagecontentsourcepolicies.yaml`, `samples-patch.yaml` and `update-graph.yaml` to the running cluster to complete the offline configuration

### 4. Keeping the mirror updated

OpenShift content is updated on a continuous basis. In order to keep your content mirror updated you can simply run `oc adm mirror` on regular basis, for instance as part of a system cronjob:

# cat /etc/crontab

0 0 * * * /usr/local/bin/oc adm mirror -f /home/openshift/mirror.yaml

Any missing or new content will be automatically added to the content mirror, whether its on disk or in a registry.

Incremental mirroring is supported for OpenShift release images and Operator Catalogs:

#### Automatically adding newer OCP releases

In the simple form, `oc adm mirror` will mirror the release of the OCP that the `oc` binary belongs to, e.g. if you are running `oc` in version v4.6.11 it would mirror all OCP release images for OCP v4.6.11.
In it's advanced form, using the mirror configuration file, you can ask it to mirror all released versions in a particular channel, e.g. `v4.7.x` in `stable-4.7`. Subsequent executions of `oc adm mirror` will automatically pick up any newly released z-stream, e.g. `v4.7.2` when they become available.

#### Automatically adding newer Operators

In the simple format, `oc adm mirror` mirrors all Operators which can lead to thousands of images being downloaded when mirroring for the first time to an empty registry or disk location. Even when subsequent mirroring runs download newer content only, the initial mirror will take up hundreds of GB of disk space and take hours to complete.
That's why it is recommended to use the mirror control file to more granularly select the Operators you wish to have available in your disconnected cluster.

When selecting based on version or opting to mirror "only the latest version" of all Operators, `oc adm mirror` only needs to download a fraction of the data. On subsequent runs, `oc adm mirror` intelligently compares which Operators are available in the mirror set and computes what needs to be downloaded to allow updates to newer versions that might have been released.

For example, at the stage when `oc adm mirror -f mirror.yaml` was run for the first time it may have been configured to mirror all the latest Operators of a catalog. For a particular Operator, e.g. `example-operator` this might have been version `v1.2`. Only the images associated with that version would be downloaded, not previous versions like `v1.1` or `v1.0`.
When `oc adm mirror` is run again at a later point in time the newest version might be `example-operator.v1.5` and in the meantime `v1.3` and `v1.4` might have been released as wellw. In this case all those versions will be mirrored as part of the incremental update, because the Operator update graph specified that the correct path from `v1.2` to `v1.5` looks like this: `v1.2 -> v1.3 -> v1.4 -> v1.5`

## Performing a initial disconnected mirror across a network air-gap

In some environments the registries that hold the offline mirror content for OpenShift are not connected to the internet either. In this case the data has to traverse an air-gap.

To keep this section brief, only the difference to the dual-homed bastion procedure are outlined. Read the previous section about **#Performing a initial disconnected mirror on a dual-homed bastion** before you proceed in this section.

### 1. Create a disk-based mirror

On a system with an internet connection use `oc adm mirror` to select which data to mirror. Refer to the previous section about options on selecting content mirror.

Mirror everything to a tar ball called *offline-bundle.tar.gz*:

`$ oc adm mirror --to-disk=offline-bundle.tar.gz`

Mirror selected content to a tar ball called *offline-bundle.tar.gz*:

`$ oc adm mirror -f mirror.yaml --to-disk=offline-bundle.tar.gz`

Copy *offline-bundle.tar.gz* to removable media, such as USB sticks or DVDs.

### 2. Transport the data to a host behind the air-gap

On a system with access to the networks in which the OpenShift cluster nodes will reside, make the offline bundle tar ball available in your working directory. A read-only mount works fine.

### 3. Prepare a registry

Similar to the dual-homed bastion setup you need a mirror registry. Refer to the previous chapter on how to either prepare that registry or use `quay-install` to quickly stand up a registry.

By default, `oc adm mirror` expected a registry running on the same node on port 443. If you did not use `quay-install` on this node or you have a registry running on another node use `oc adm mirror login --registry=<fqdn>:<port>` to authenticate. Also append `--registry=<fqdn>:<port>` to all of the following commands.

### 4. Load the offline bundle into a registry

Use the following command to load the offline bundle data into your registry:

`$ oc adm mirror --from-disk=offline-bundle.tar.gz`

At this point you have the same state as if your created the offline mirror on a dual-homed bastion host. You may now proceeed with installing your cluster:

### 5. Start the disconnected installation

Iinstruct `openshift-install` to refer to the offline mirror directory created by the previous step:

`$ openshift-install create cluster --mirror-config=./offline-manifests/`