# Generic CCM Testing in Kubernetes
Capturing some thoughts and ideas about creating a set of generic CCM tests for use by the wider cloud provider community. These tests will be owned by SIG Cloud Provider and be used as a baseline for ensuring that all external CCMs have a minimal signal for testing.

## Background
For several years now (see [kubernetes issue#70194](https://github.com/kubernetes/kubernetes/issues/70194)) there has been a desire to remove the provider specific details which exist in the main kubernetes repository.

Currently, the tests in the `test/e2e/cloud` and `test/e2e/network` which exercise functionality of the cloud controllers are gated by logic which detects the underlying infrastructure platform. Some of these tests also contain instructions which can only be executed on those specific platforms. This creates a situation where the tests in the `kubernetes/kubernetes` repository have a direct dependency on the specific cloud provider platforms.

## Ideas
These are some ideas captured after reviewing the available material, and talking with members of SIG Cloud Provider, SIG Storage, and SIG Testing to help drive the conversation forward towards creating a set of generic cloud controller tests in kubernetes.

* Create a new set of tests, inspired by the current tests, under the `test/e2e/cloud/external` directory to capture the desired generic behavior.
* Utilize a configuration file similar to the one used by the SIG Storage CSI tests (see this [README](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/external/README.md)) to allow individual cloud providers to signal which capabilities their CCMs implement.
* Create an interface which will live in the `test/e2e/cloud/external` directory to allow individual cloud providers to implement logic necessary for tests which require infrastructure interactions (e.g. deleting Nodes).
* Promote the implementation of this interface to individual cloud providers so that they can run the generic tests.
* Create a mock provider that can be used during configuration or in cases where there is no specific platform. This could live in the `kubernetes/cloud-provider` repository.
* Use selective compilation during CI testing to allow inclusion of infrastructure provider implementations.

## Open Questions
* Does the current CI testing workflow allow for a conditional build which would change dependencies based on the infrastructure platform being tested?
* If so, how does this work?
* may not be done or standardized currently, but it is being done currently. perhaps look at sig storage tests.
* it's defined in how prow job are setup, cluster info comes from there. (need more investigation about provisioners, boskos resources, might need some standarization here)
* Is there a better way to implement provider specific functionality than creating a common interface?
* like the idea of a contract that should be implemented, enforcing at a type level is good. but importing based on provider conditions may be problematic.
* Should we do something different, such as inverting the proposed pattern to allow invididual provider repositories to include the kubernetes tests in their vendored dependencies for local testing?
* inverting might be preferred given that then we won't be importing from k/k.
* this raises the question of what version should the cloud-providers repos import? can this be done dynamically?
* maybe look at conformance tests to see how they are done, and how we gate the platforms and builds there.
* look at cluster-api for configuration advice, look at e2e tests.
* For tests that require functionality which is only implemented by some cloud providers, should those tests live in another repository (e.g. `kubernetes/cloud-provider`)?
* perhaps needs more investigation
* want to look into who owns the process of running specific provider tests for the community