CNF Conformance Beta Test (#1) Instructions

# CNF Conformance Beta Test (#1) Instructions  Last-updated: 2020-06-05  ---- ## Intro Thanks again for volunteering to test the CNF Conformance Test Suite! The average time to run the test suite and fill out the questionnaire is around 90 minutes. Feel free to open the [Google Forms questionnaire](https://forms.gle/MWUZZVR7HUcTtVpm6) while you follow the beta test instructions. To file bugs, request new tests, request new features and review any known issues, please see the [FAQ section](#FAQ). If you have any questions, please post a note in [#cnf-conformance-dev](https://cloud-native.slack.com/archives/C014TNCEX8R/p1591391228000600) slack channel. We kindly request that all questionnaires are completed by 11:59pm UTC on Monday, June 22nd. Let's get started!  ## Requirements Each test user will provide their own test workstation for running the CNF Conformance test software. Access to a managed K8s cluster used as the test environment will be provided. #### Test Workstation requirements - Supported operating system: - Linux: x86, 64bit. Kernel v4+. Example distros: - Ubuntu 18.04 LTS - Windows: WSL - Ubuntu 18.04 & 20.04 LTS - [Kubectl binary is installed](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on test workstation: - Type `which kubectl` to see if you have kubectl installed already. You should see some output message with the path to kubectl, if not it will be blank. - K8s access configuration file (aka kubeconfig) - Provided to you - Terminal application for CLI usage. Examples - Linux: xterm - Windows: bash.exe (from WSL) ## Steps for Testing **Start by opening a Terminal window** _Note: Everything will be run from a Terminal window (CLI/shell prompt)_ ### 1. Download + install the conformance suite - Create folder used for testing: - `mkdir -p ~/betatest1` - Go into the folder: - `cd ~/betatest1` - Download the [v0.7.2-beta1](https://github.com/cncf/cnf-conformance/releases/tag/v0.7.2-beta1) binary into the folder - `wget -O cnf-conformance https://github.com/cncf/cnf-conformance/releases/download/v0.7.2-beta1/cnf-conformance-v0.7.2-amd64-static` - Make the binary executable (eg. chmod +x cnf-conformance) - `chmod +x cnf-conformance` - Download the [scoring configuration](https://github.com/cncf/cnf-conformance/blob/release-v0.7-beta1/points.yml) by running: - `wget -O points.yml https://raw.githubusercontent.com/cncf/cnf-conformance/release-v0.7-beta1/points.yml` ### 2. Accessing your managed K8s test environment [Access](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/) to a managed K8s cluster will be via a [KUBECONFIG environment variable](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable). (See [K8s Getting started guide](https://kubernetes.io/docs/setup/) for more) - Download the K8s kubeconfig file to your workstation: 1. Copy the URL for the kubeconfig from your email. 1. Download the kubeconfig file using the URL - `wget -O betatest1.kubeconfig` <PASTE URL FROM EMAIL HERE> - Set the KUBECONFIG environment variable, by typing: - `export KUBECONFIG=$(pwd)/betatest1.kubeconfig` - Validate access to the cluster: - Run `kubectl cluster-info` - Kubernetes master node URL in output should match the URL in your welcome email (and kubeconfig file): - ie. https://111.22.33.444:5555 ### 3. Initialization, pre-reqs - Initialize the test suite by running: - `./cnf-conformance setup` - Output message should say `Setup complete` This will create the working folder in your test directory, which downloads and sets up depedencies for CNF Conformance. ### 4. CNF Setup [CoreDNS](https://github.com/coredns/coredns) will be used as the test CNF. - Download the conformance configuration to test CoreDNS: - `wget -O cnf-conformance.yml https://raw.githubusercontent.com/cncf/cnf-conformance/release-v0.7-beta1/example-cnfs/coredns/cnf-conformance.yml` - Prepare the test suite to use the CNF by running: - `./cnf-conformance cnf_setup cnf-config=./cnf-conformance.yml` ### 5. Running the test with example CNF #### 5a. All tests - Run all of the workload tests against the CNF: - `./cnf-conformance all` - Type `Y` to confirm overwriting the existing results.yml - Review results from test. cnf-conformance will display the output of the results yml file. #### 5b. Single category test - Run the following command to initialize a new results.yml: - `./cnf-conformance setup` - Type `Y` to confirm overwriting the existing results.yml - Run workload tests from a single category against the CNF: - `./cnf-conformance microservice` - Review results from test. #### 5c. Single test - Run the following command to initialize a new results.yml: - `./cnf-conformance setup` - Type `Y` to confirm overwriting the existing results.yml - Run a single workload test against the CNF: - `./cnf-conformance nodeport_not_used` - Review results from test. ### 6. Cleanup - Run the following to clean up the installed cnf: `./cnf-conformance cnf_cleanup cnf-config=./cnf-conformance.yml` - You can now safely remove the temporary test directory (eg. `betatest`) ### Feedback from testing Please provide feedback in the [Beta test questionnaire Google Form]( https://forms.gle/MWUZZVR7HUcTtVpm6). ### FAQ - **How to report a bug** :bug: - If you would like to report a bug, please create a [new issue](https://github.com/cncf/cnf-conformance/issues/new?assignees=&labels=bug&template=bug-report.md&title=%5BBUG%5D) (using the **Bug Report** Template) - **How to request a new workload test** :new: - If you would like to request a new workload test, please create a [new issue](https://github.com/cncf/cnf-conformance/issues/new?assignees=&labels=workload&template=new-workload-test.md&title=%5BWorkload%5D) (using the **New Workload Test** Template) - **How to request a new platform test** :new: - If you would like to request a new platform test, please create a [new issue](https://github.com/cncf/cnf-conformance/issues/new?assignees=&labels=platform&template=new-platform-test.md&title=%5BPlatform%5D) (using the **New Platform Test** Template) - **How to request a new feature** :new: - If you would like to request an enhancement, please create a [new issue](https://github.com/cncf/cnf-conformance/issues/new?assignees=&labels=enhancement&template=feature-request.md&title=%5BFeature%5D) (using the **Feature Request** Template) - **Scores** :pencil: :-1: :+1: - Why is the max score greater than the total number of tests? - Tests can have different points assigned. See points.yml - **Messages about overwriting the results.yml.** :writing_hand: :recycle: - Why does the CLI keep complaining about overwriting results.yml. - Handling creation and backup of results.yml are still in progress. Some commands (like `all`) take care of backing up at the end while others do not handle creation or renaming it. - **On WSL in Windows, I'm getting ERROR on .wget-hsts being not readable?** :red_circle: - This seems to be permissions due to the default umask env settings when wget creates the file `.wget-hsts` if not present in your users home dir. - See [Github WSL #352](https://github.com/microsoft/WSL/issues/352) for more details. - Workaround Steps: - After launching `bash.exe` do the following: - `umask 0022` - If your user already has a ~/.wget-hsts file, simply: `chmod 644 ~/.wget-hsts` to remove future errors. - **I don't have kubectl installed on WSL, is there an easy way to install it?** :arrow_right: - You can install the latest stable release by doing the following: - `curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl` - `chmod +x kubectl` - Then move it so the command is in your current PATH: `mv kubectl /usr/local/bin` - **How can I build the binary from source for the beta test?** :building_construction: - Install Crystal https://crystal-lang.org/install/ - Clone the repo `git clone https://github.com/cncf/cnf-conformance.git` - Go into the repo directory: `cd cnf-conformance` - Checkout the [release-v0.7-beta1](https://github.com/cncf/cnf-conformance/tree/release-v0.7-beta1) release: `git checkout release-v0.7-beta1` - Build the binary: `crystal build --release src/cnf-conformance.cr` - Copy the binary to your test directory (eg. ~/betatest1): `cp cnf-conformance ~/betatest1` - Go back to the test directory and use the new binary. (eg. `cd ~/betatest1`) - **How can I build a binary with the latest changes?** :new: :building_construction: - See https://github.com/cncf/cnf-conformance/blob/master/INSTALL.md#source-install - **What other documentation is available?** :book: - [USAGE guide](https://github.com/cncf/cnf-conformance/blob/master/USAGE.md) - all available commands - [INSTALL guide](https://github.com/cncf/cnf-conformance/blob/v0.7.0-beta1/INSTALL.md) - source and binary installation - [Test Categories](https://github.com/cncf/cnf-conformance/blob/master/TEST-CATEGORIES.md) - goals for each test category - [README](https://github.com/cncf/cnf-conformance/blob/master/README.md) - CNF Conformance overview