# ClusterAPI - Running e2e tests & controllers locally with Intellij

This document contains some additional notes for the corresponding Zoom session ([link](https://vmware.zoom.us/rec/share/22e9r0UGbwkWUl59cLIyG2bS2hIsPr1Di2r6SfdTT07b64HKyxXuJeA3vvQ7uww.4-GHynbnQMHIEB1g) Passcode: `N20+nkQ%`).

Overall we will:
* Run the CAPD-based quickstart e2e test via Intellij
* Run the capi-controller via Intellij and proxy webhook requests from the kind cluster to the locally running controller

Note:
* There is already documentation about how to run e2e tests via an IDE in the [CAPI book](https://cluster-api.sigs.k8s.io/developer/testing.html#test-execution-via-ide). The current doc is a concrete walkthrough and additionally contains instructions on how to run a controller locally.

# Prerequisites

* The usual CAPD dependencies: kubectl, kind, Docker [link](https://cluster-api.sigs.k8s.io/user/quick-start.html#common-prerequisites)
* https://github.com/kubernetes-sigs/cluster-api repo must be cloned (later referred to as $CAPI_HOME)
* [Telepresence](https://www.telepresence.io/)
* [chrischdi/k8s-ctx-import](https://github.com/chrischdi/k8s-ctx-import) (or some other way to import the kubeconfig into the default kubeconfig)

# Running the quickstart e2e test locally

(The following is extracted from $CAPI_HOME/scripts/ci-e2e.sh)

Building the images:

cd $CAPI_HOME
export REGISTRY=gcr.io/k8s-staging-cluster-api
export PULL_POLICY=IfNotPresent

make docker-build
make -C test/infrastructure/docker docker-build

Note: This is only required if local versions of the controllers should be used, per default the daily images published under the `main` tag are used.

Generating cluster-templates:

make -C test/e2e cluster-templates

Run the e2e test via `Debug` in Intellij and place a breakpoint [here](https://github.com/kubernetes-sigs/cluster-api/blob/947147213c525b03c146f46c2eb41590abc1bf71/test/e2e/quick_start.go#L104)

Note: The following Run/Debug configuration is used:

<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="capi e2e: quickstart" type="GoTestRunConfiguration" factoryName="Go Test" folderName="test/e2e">
    <module name="cluster-api" />
    <working_directory value="$PROJECT_DIR$/test/e2e" />
    <parameters value="-e2e.config=$PROJECT_DIR$/test/e2e/config/docker.yaml -ginkgo.focus=&quot;\[PR-Blocking\]&quot; -ginkgo.v=true" />
    <envs>
      <env name="ARTIFACTS" value="$PROJECT_DIR$/_artifacts" />
    </envs>
    <kind value="PACKAGE" />
    <package value="sigs.k8s.io/cluster-api/test/e2e" />
    <directory value="$PROJECT_DIR$" />
    <filePath value="$PROJECT_DIR$" />
    <framework value="gotest" />
    <pattern value="^\QTestE2E\E$" />
    <method v="2" />
  </configuration>
</component>

Note: To run another test the `ginkgo.focus` parameter can be adjusted.

We now have a Management and a Workload cluster running.

# Running the capi-controller locally

Now we will use Telepresence to run the capi-controller locally. Telepresence will be used to proxy the webhook trafic to our local controller (roughly like [this](https://www.telepresence.io/docs/latest/concepts/intercepts/?intercept=global)).

Import the kubeconfig of the kind Management cluster

kind get kubeconfig --name=test-q5tlzb | k8s-ctx-import

Deploy telepresence:

telepresence connect

Disable the currently deployed capi-controller and its probes:

controller=capi
kubectl -n ${controller}-system patch deployment ${controller}-controller-manager --type json -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/readinessProbe"},{"op": "remove", "path": "/spec/template/spec/containers/0/livenessProbe"},{"op": "replace", "value": "k8s.gcr.io/pause:3.5", "path": "/spec/template/spec/containers/0/image"},{"op": "replace", "value": ["/pause"], "path": "/spec/template/spec/containers/0/command"}]'

Intercept webhook traffic via telepresence

telepresence intercept -n ${controller}-system ${controller}-controller-manager --port 9443

Get the webhook certificates

mkdir -p /tmp/webhook-cert
kubectl -n ${controller}-system get secret ${controller}-webhook-service-cert -o json | jq '.data."tls.crt"' -r | base64 -d > /tmp/webhook-cert/tls.crt
kubectl -n ${controller}-system get secret ${controller}-webhook-service-cert -o json | jq '.data."tls.key"' -r | base64 -d > /tmp/webhook-cert/tls.key

Start the controller via Intellij

Note: The following Run/Debug configuration is used:

<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="tele: capi controller /tmp/webhook-cert" type="GoApplicationRunConfiguration" factoryName="Go Application" folderName="CAPI">
    <module name="cluster-api" />
    <working_directory value="$PROJECT_DIR$/" />
    <parameters value="--webhook-cert-dir=/tmp/webhook-cert --feature-gates=MachinePool=true,ClusterResourceSet=true,ClusterTopology=true" />
    <envs>
      <env name="CAPI_MAC_FIX_REST_CONFIG" value="true" />
    </envs>
    <kind value="PACKAGE" />
    <package value="sigs.k8s.io/cluster-api" />
    <directory value="$PROJECT_DIR$" />
    <filePath value="$PROJECT_DIR$/main.go" />
    <method v="2" />
  </configuration>
</component>

Now you can play around by setting breakpoints in the controller and modifying the CAPI resources!

# Running the capi-controller locally (Part II - MacOS hack)

At least on MacOS the capi-controller will frequently log errors because because it cannot reach the apiservers of the workload clusters. That's more or less the same issue as documented [here](https://cluster-api.sigs.k8s.io/clusterctl/developers.html?highlight=macos#fix-kubeconfig-when-using-docker-on-macos) just this time **inside** the CAPI controller (tl;dr the workload kubeconfig is not valid when using it locally).

To workaround this, the Run/Debug configuration above sets the `CAPI_MAC_FIX_REST_CONFIG` env var and the following patch must be applied:

Index: controllers/remote/cluster.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/controllers/remote/cluster.go b/controllers/remote/cluster.go
--- a/controllers/remote/cluster.go	(revision HEAD)
+++ b/controllers/remote/cluster.go	(revision Staged)
@@ -18,6 +18,9 @@controllers/remote/cluster.go
 
 import (
 	"context"
+	"fmt"
+	"os"
+	"strings"
 	"time"
 
 	"github.com/pkg/errors"
@@ -59,8 +62,33 @@
 		return nil, errors.Wrapf(err, "failed to create REST configuration for Cluster %s/%s", cluster.Namespace, cluster.Name)
 	}
 
+	if os.Getenv("CAPI_MAC_FIX_REST_CONFIG") != "" {
+		lbContainerName := cluster.Name + "-lb"
+		port, err := findLoadBalancerPort(ctx, lbContainerName)
+		if err != nil {
+			return nil, errors.Wrapf(err, "failed to get lb port")
+		}
+		restConfig.Host = fmt.Sprintf("https://127.0.0.1:%s", port)
+		restConfig.Insecure = true
+		restConfig.CAData = nil
+	}
+
 	restConfig.UserAgent = DefaultClusterAPIUserAgent(sourceName)
 	restConfig.Timeout = defaultClientTimeout
 
 	return restConfig, nil
 }
+
+func findLoadBalancerPort(ctx context.Context, lbContainerName string) (string, error) {
+	portFormat := `{{index (index (index .NetworkSettings.Ports "6443/tcp") 0) "HostPort"}}`
+	getPathCmd := NewCommand(
+		WithCommand("docker"),
+		WithArgs("inspect", lbContainerName, "--format", portFormat),
+	)
+	stdout, _, err := getPathCmd.Run(ctx)
+	if err != nil {
+		return "", err
+	}
+
+	return strings.TrimSpace(string(stdout)), nil
+}
Index: controllers/remote/command.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/controllers/remote/command.go b/controllers/remote/command.go
new file mode 100644
--- /dev/null	(revision Staged)
+++ b/controllers/remote/command.go	(revision Staged)
@@ -0,0 +1,101 @@
+/*
+Copyright 2019 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package remote implements command execution functionality.
+package remote
+
+import (
+	"context"
+	"io"
+	"os/exec"
+
+	"github.com/pkg/errors"
+)
+
+// Command wraps exec.Command with specific functionality.
+// This differentiates itself from the standard library by always collecting stdout and stderr.
+// Command improves the UX of exec.Command for our specific use case.
+type Command struct {
+	Cmd   string
+	Args  []string
+	Stdin io.Reader
+}
+
+// Option is a functional option type that modifies a Command.
+type Option func(*Command)
+
+// NewCommand returns a configured Command.
+func NewCommand(opts ...Option) *Command {
+	cmd := &Command{
+		Stdin: nil,
+	}
+	for _, option := range opts {
+		option(cmd)
+	}
+	return cmd
+}
+
+// WithStdin sets up the command to read from this io.Reader.
+func WithStdin(stdin io.Reader) Option {
+	return func(cmd *Command) {
+		cmd.Stdin = stdin
+	}
+}
+
+// WithCommand defines the command to run such as `kubectl` or `kind`.
+func WithCommand(command string) Option {
+	return func(cmd *Command) {
+		cmd.Cmd = command
+	}
+}
+
+// WithArgs sets the arguments for the command such as `get pods -n kube-system` to the command `kubectl`.
+func WithArgs(args ...string) Option {
+	return func(cmd *Command) {
+		cmd.Args = args
+	}
+}
+
+// Run executes the command and returns stdout, stderr and the error if there is any.
+func (c *Command) Run(ctx context.Context) ([]byte, []byte, error) {
+	cmd := exec.CommandContext(ctx, c.Cmd, c.Args...) //nolint:gosec
+	if c.Stdin != nil {
+		cmd.Stdin = c.Stdin
+	}
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		return nil, nil, errors.WithStack(err)
+	}
+	stderr, err := cmd.StderrPipe()
+	if err != nil {
+		return nil, nil, errors.WithStack(err)
+	}
+	if err := cmd.Start(); err != nil {
+		return nil, nil, errors.WithStack(err)
+	}
+	output, err := io.ReadAll(stdout)
+	if err != nil {
+		return nil, nil, errors.WithStack(err)
+	}
+	errout, err := io.ReadAll(stderr)
+	if err != nil {
+		return nil, nil, errors.WithStack(err)
+	}
+	if err := cmd.Wait(); err != nil {
+		return output, errout, errors.WithStack(err)
+	}
+	return output, errout, nil
+}