# Enable Private Hypershift Clusters

## Design

If public, management cluster default router and public DNS can be used. This is how Hypershift currently works.

If private, all communication with the guest cluster must occur over [PrivateLink](https://aws.amazon.com/privatelink/).

PrivateLink the a combination of an Endpoint Service in the provider VPC (i.e. management cluster VPC) and an Endpoint in the consumer VPC (i.e. the guest cluster VPC). Only NLBs are supported as Endpoint Service targets. The Endpoints will be [Interface Endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-interface.html).

Ingress controller supports creating a Service with the [NLB](https://github.com/openshift/api/blob/c9d6fcc62893a426618b74c3ce415e5ae045f7e6/operator/v1/types_ingress.go#L373-L405) annotation.

A private topology requires the following AWS resources in the management VPC:

Per HostedCluster in the management cloud account / cluster:
* An internal IngressController for the hosted control plane configured to create an NLB (what is the `domain` of this IngressController)
* A `*-private` Route for each service with publishing type Route
* A `kube-apiserver-private` Service of type LoadBalancer an annotated to create an NLB
* An Endpoint Service that points to the internal ingress NLB
* An Endpoint Service that points to the internal KAS NLB


Per HostedCluster in the guest account:
* An Endpoint for the KAS Endpoint Service published from the management VPC
* An Endpoint for the ingress Endpoint Service published from the management VPC

**ISSUE** This introduces a new ordering which seems problematic. The Endpoints created in the guest VPC should be created by the CLI at `create infra` time. However, the Endpoints in the guest account can not be created until the Endpoint Services exist in the management VPC. The Endpoint Services can not exist until the KAS and ingress NLBs exist. Those don't exist until the HostedCluster exists. The HostedCluster needs inputs (vpc, subnet,etc) from `create infra`. Cycle found.


Endpoints are per VPC and thus not sharable amoung multiple guest clusters in the same account.

Service Endpoint (without private DNS and thus no domain ownership verification) are quick to provision. Endpoints in the guest VPC are slower.

There is an option for adding private DNS to the guest account when adding the Endpoint based on the Endpoint Service configuration. However, this delays the Endpoint Service creation time as DNS ownership has to be confirmed by adding a TXT record with a particular value to the DNS zone.

The end result would be that for services with a Route publishing type, the host address would be the DNS name of an A record whose value is the private IP of the Endpoint interface pointing to the HCP private ingress Endpoint Service published from the management account.

**ISSUE** One issue to be resolved is when the ingress endpoint when the mgmt cluster and guest cluster are in the same account. In this case, the ingress-operator in the mgmt cluster manages the wildcard DNS record for the private IngressController in the private zone for the mgmt cluster. The guest must use the Route's `host` to connect or the router will not be able to match the Route. However, the ingress-operator configures the wildcard record to point to the NLB in the management account, assuming anyone resolving the name will be connecting from the from within the VPC, not from another potentially disconnected VPC in the same account, which will use the same private DNS zone managed by the ingress-operator in the mgmt cluster.


For the KAS service with publishing type LoadBalancer, the host address would be the DNS name of an A record whose value is the private IP of the Endpoint interface pointing to the non-shared KAS Endpoint Service published from the management account.

Endpoint Services also have an acceptance policy that determines whether an accpetance action is required from the provider side in order for an Endpoint to be created in another account.

type InfrastructureStatus struct {
	APIHost                 string
	OAuthHost               string
	KonnectivityHost        string
	OpenShiftAPIHost        string
	OauthAPIServerHost      string
	PackageServerAPIAddress string
}

These are the services for which we need to know the address before we create the control plane and guest cluster manifests.

All have a servicePublishingStrategy of type `Route` on AWS except the APIServer (kube-apiserver) which is type `LoadBalancer`.

Today, the `APIHost` is determined by creating the kube-apiserver Service of type LoadBalancer and waiting for the status to reflect the DNS name of ELB.

The rest of the hostnames are determined by creating Routes and then reading back the `spec.host`.

## Manifests

`clusters-example-private` IngressController

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: clusters-example-private
  namespace: openshift-ingress-operator
spec:
  replicas: 2
  domain: clusters-example.sjenning.hypershift.sjennings.me
  routeSelector:
    matchLabels:
      hypershift.openshift.io/hosted-control-plane: clusters-example
  endpointPublishingStrategy:
    loadBalancer:
      scope: Internal
      providerParameters:
        type: AWS
        aws:
          type: NLB
          networkLoadBalancer: {}

Patch to default ingress to not admit private routes

spec:
  routeSelector:
    matchExpressions:
    - key: hypershift.openshift.io/hosted-control-plane
      operator: DoesNotExist

`kube-apiserver-private` Service resulting in NLB

apiVersion: v1
kind: Service
metadata:
  name: kube-apiserver-private
  namespace: clusters-example
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
  ports:
  - port: 6443
    protocol: TCP
    targetPort: 6443
  selector:
    app: kube-apiserver
  type: LoadBalancer

Example private ignition-server Route

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    hypershift.openshift.io/hosted-control-plane: clusters-example
  annotations:
    hypershift.openshift.io/cluster: clusters/example
  name: ignition-server-private
  namespace: clusters-example
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: ignition-server

## Scenarios

### Public only

### Private only

### Public/Private