# operator-sdk scorecard notes
###### tags: `scorecard`

## scorecard

* https://v0-19-x.sdk.operatorframework.io/docs/
* https://v0-19-x.sdk.operatorframework.io/docs/scorecard/custom-tests/

* walk through creating operators through testing
* https://labs.consol.de/development/2021/05/03/helm_based_operator.html

## env test
* Testing the operator w/ EnvTest
* https://sdk.operatorframework.io/docs/building-operators/golang/testing/
* https://book.kubebuilder.io/cronjob-tutorial/writing-tests.html
*

## kuttl

* https://sdk.operatorframework.io/docs/testing-operators/scorecard/kuttl-tests/

## GOAL - execute OADP e2e against downstream build via scorecard
* https://github.com/openshift/oadp-operator/tree/master/tests/e2e

## Holy Crap this works
* https://github.com/AICoE/peak
* Custom Fork: required for missing libs
* https://github.com/weshayutin/peak
* diff the dockerfile

* Tests are defined here:
* https://github.com/weshayutin/rad-spark-sample-tests
#### setup

git clone https://gitlab.cee.redhat.com/mig-integration/scorecard-peak.git
cd scorecard-peak
git submodule update --init
git submodule update --remote --merge

#### build
in the scorecard-peak directory

podman build . -t quay.io/rhn_engineering_whayutin/peaks2i
podman push quay.io/rhn_engineering_whayutin/peaks2i
oc create secret docker-registry secret-push --docker-server=quay.io --docker-username=rhn_engineering_whayutin --docker-password=<snip> --docker-email=[e-mail]
oc secrets link default secret-push --for=pull --namespace openshift-adp
oc new-build quay.io/rhn_engineering_whayutin/peaks2i~https://github.com/weshayutin/rad-spark-sample-tests --strategy=source --to-docker=true --to=quay.io/rhn_engineering_whayutin/testimage
podman run --rm -t quay.io/rhn_engineering_whayutin/testimage usage

some updates

podman build . -t quay.io/rhn_engineering_whayutin/oadp_peaks2i
podman push quay.io/rhn_engineering_whayutin/oadp_peaks2i:latest
oc create secret generic quay --from-file=.dockerconfigjson=/home/whayutin/.docker/config.json --type=kubernetes.io/dockerconfigjson
oc new-build quay.io/rhn_engineering_whayutin/oadp_peaks2i~https://github.com/weshayutin/rad-spark-sample-tests --strategy=source --to-docker=true --to=quay.io/rhn_engineering_whayutin/oadp_testimage --push-secret=quay
podman run --rm -t quay.io/rhn_engineering_whayutin/oadp_testimage usage

### it's missing the rpm ncurses, tput is from ncureses

## WOOT

### scorecard config

  - entrypoint:
    - scorecard
    image: quay.io/rhn_engineering_whayutin/testimage
    labels:
      suite: peak
    storage:
      spec:
        mountPath: {}

### setup service account

oc create sa wes1 -n openshift-adp
oc policy add-role-to-user admin system:serviceaccount:openshift-adp:wes1 -n openshift-adp

FULL ADMIN

oc create sa wes1
oc policy add-role-to-user admin system:serviceaccount:wes1

### Execute the scorecard test

operator-sdk scorecard bundle   --selector=suite=peak -w "300s" --kubeconfig /home/whayutin/.agnosticd/wdh1109ocp4a/ocp4-cluster_wdh1109ocp4a_kubeconfig -s wes1 -n openshift-adp

* Result:

[whayutin@thinkdoe oadp-operator-bundle]$ operator-sdk scorecard bundle   --selector=suite=peak -w "3000s" --kubeconfig /home/whayutin/.agnosticd/wdh0104pelorus1/ocp4-cluster_wdh0104pelorus1_kubeconfig  -s wes1 -n openshift-adp
--------------------------------------------------------------------------------
Image:      quay.io/rhn_engineering_whayutin/oadp_testimage:stable1
Entrypoint: [scorecard]
Labels:
	"suite":"peak"
Results:
	Name: rad-spark-sample-tests
	State: pass

	Suggestions:
		All tests passed. Use 'base64 -d' to view the logs.
	Errors:
		
	Log:
		cGFzc2VkOiByYWQtc3Bhcmstc2FtcGxlLXRlc3RzL3Rlc3Quc2gKL29wdC9wZWFrL3J1bi5zaCB0b29rIDEgc2Vjb25kcwpbSU5GT10gRXhpdGluZyB3aXRoIDAK

### decode log message
* using https://www.base64decode.org/ because base64 -d aint doing it for me.

passed: rad-spark-sample-tests/test.sh
/opt/peak/run.sh took 1 seconds
[INFO] Exiting with 0

## local / private s2i build

## ansible vault notes

https://www.digitalocean.com/community/tutorials/how-to-use-vault-to-protect-sensitive-ansible-data-on-ubuntu-16-04

## Design

* Use the scorecard compliant dockerfile from [scorecare peak](https://gitlab.cee.redhat.com/mig-integration/scorecard-peak)
* add requirements for ansible, ginko etc.

* Add ansible that execute e2e to peak [test.sh](https://github.com/weshayutin/rad-spark-sample-tests/blob/master/test.sh)
* Ensure the s3 bucket for e2e is encrypted with vault
* Include the value secret in the test.sh ( internal repo )
* Push peak container to quay

Questions:
* Do other groups use scorecard for integration tests? If so how?
* ANSWER: not really.. only operators outside of openshift would need this, and they are not at this time

* Is it possible to push the scorecard container to internal only repo?
* ANSWER: RFE: https://github.com/operator-framework/operator-sdk/issues/5339

  sorry meant to say file the issue, THEN link it to the RFE meta issue.
1:24
since some RFEs need more details.

*
* Is there an alternative to scorecard peak?
* Has anyone attempted to execute ginko e2e tests via scorecard?
* Answer: not at this time, no

## Isuse Tracker for RFE's
https://issues.redhat.com/browse/OSDK-2114

## Docs on custom scorecard tests
* https://docs.openshift.com/container-platform/4.9/operators/operator_sdk/osdk-scorecard.html#osdk-scorecard-custom-tests_osdk-scorecard

# Do over
https://sdk.operatorframework.io/docs/testing-operators/scorecard/custom-tests/

## scorecard config yaml

 38   - entrypoint:
 39     - scorecard-test
 40     - customtest1
 41     image: quay.io/rhn_engineering_whayutin/oadp_scorecard:9
 42     labels:
 43       suite: foo
 44       test: customtest1
 45     storage:
 46       spec:
 47         mountPath: {}
 48   - entrypoint:
 49     - scorecard-test
 50     - customtest2
 51     image: quay.io/rhn_engineering_whayutin/oadp_scorecard:9
 52     labels:
 53       suite: foo
 54       test: customtest2
 55     storage:
 56       spec:
 57         mountPath: {}

OR

  - entrypoint:
    - oadp-scorecard-test
    - customtest1
    image: quay.io/rhn_engineering_whayutin/oadp_scorecard_oc:8
    labels:
      suite: foo
      test: customtest1
    storage:
      spec:
        mountPath: {}
  - entrypoint:
    - oadp-scorecard-test
    - customtest2
    image: quay.io/rhn_engineering_whayutin/oadp_scorecard_oc:8
    labels:
      suite: foo
      test: customtest2
    storage:
      spec:
        mountPath: {}

## Result:

[whayutin@thinkdoe oadp-operator-bundle]$ operator-sdk scorecard -x  bundle -o text  --selector=suite=foo -w "3000s" --kubeconfig /home/whayutin/.agnosticd/wdh0104pelorus1/ocp4-cluster_wdh0104pelorus1_kubeconfig    -s wes1 -n openshift-adp --verbose
DEBU[0000] Debug logging is set                         
--------------------------------------------------------------------------------
Image:      quay.io/rhn_engineering_whayutin/oadp_scorecard:9
Entrypoint: [scorecard-test customtest1]
Labels:
	"test":"customtest1"
	"suite":"foo"
Results:
	Name: customtest1
	State: pass



--------------------------------------------------------------------------------
Image:      quay.io/rhn_engineering_whayutin/oadp_scorecard:9
Entrypoint: [scorecard-test customtest2]
Labels:
	"suite":"foo"
	"test":"customtest2"
Results:
	Name: customtest2
	State: pass

## DockerFile

  1 FROM quay.io/operator-framework/scorecard-test:v1.16
  2 COPY oadp-scorecard-test /usr/local/bin/oadp-scorecard-test
  3 ENV HOME=/opt/scorecard-test USER_NAME=scorecard-test USER_UID=1001

:::info
The dockerfile and scorecard config work together where the entrypoint parameter [0] is the binary name, in this case "oadp-scorecard-test" and [1] is the testname or "entrypoint := os.Args[1:]" from the script
:::

## Execute
* locally w/

podman run --user 1001 -ti --entrypoint=oadp-scorecard-test -v $(pwd)/bundle:/bundle:Z quay.io/rhn_engineering_whayutin/oadp_scorecard_oc:7 customtest1

* on cluster

 operator-sdk scorecard -x  bundle -o text  --selector=suite=foo -w "3000s" --kubeconfig /home/whayutin/.agnosticd/wdh0104pelorus1/ocp4-cluster_wdh0104pelorus1_kubeconfig    -s wes1 -n openshift-adp --verbose

## source go file in this example

// Copyright 2020 The Operator-SDK Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"os"

	scapiv1alpha3 "github.com/operator-framework/api/pkg/apis/scorecard/v1alpha3"
	apimanifests "github.com/operator-framework/api/pkg/manifests"
)

// This is the custom scorecard test example binary
// As with the Redhat scorecard test image, the bundle that is under
// test is expected to be mounted so that tests can inspect the
// bundle contents as part of their test implementations.
// The actual test is to be run is named and that name is passed
// as an argument to this binary.  This argument mechanism allows
// this binary to run various tests all from within a single
// test image.

const PodBundleRoot = "/bundle"

func main() {
	entrypoint := os.Args[1:]
	if len(entrypoint) == 0 {
		log.Fatal("Test name argument is required")
	}

	// Read the pod's untar'd bundle from a well-known path.
	cfg, err := apimanifests.GetBundleFromDir(PodBundleRoot)
	if err != nil {
		log.Fatal(err.Error())
	}

	var result scapiv1alpha3.TestStatus

	// Names of the custom tests which would be passed in the
	// `operator-sdk` command.
	switch entrypoint[0] {
	case CustomTest1Name:
		result = CustomTest1(cfg)
	case CustomTest2Name:
		result = CustomTest2(cfg)
	default:
		result = printValidTests()
	}

	// Convert scapiv1alpha3.TestResult to json.
	prettyJSON, err := json.MarshalIndent(result, "", "    ")
	if err != nil {
		log.Fatal("Failed to generate json", err)
	}
	fmt.Printf("%s\n", string(prettyJSON))

}

// printValidTests will print out full list of test names to give a hint to the end user on what the valid tests are.
func printValidTests() scapiv1alpha3.TestStatus {
	result := scapiv1alpha3.TestResult{}
	result.State = scapiv1alpha3.FailState
	result.Errors = make([]string, 0)
	result.Suggestions = make([]string, 0)

	str := fmt.Sprintf("Valid tests for this image include: %s %s",
		CustomTest1Name,
		CustomTest2Name)
	result.Errors = append(result.Errors, str)
	return scapiv1alpha3.TestStatus{
		Results: []scapiv1alpha3.TestResult{result},
	}
}

const (
	CustomTest1Name = "customtest1"
	CustomTest2Name = "customtest2"
)

// Define any operator specific custom tests here.
// CustomTest1 and CustomTest2 are example test functions. Relevant operator specific
// test logic is to be implemented in similarly.

func CustomTest1(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus {
	r := scapiv1alpha3.TestResult{}
	r.Name = CustomTest1Name
	r.State = scapiv1alpha3.PassState
	r.Errors = make([]string, 0)
	r.Suggestions = make([]string, 0)
	almExamples := bundle.CSV.GetAnnotations()["alm-examples"]
	if almExamples == "" {
		fmt.Println("no alm-examples in the bundle CSV")
	}

	return wrapResult(r)
}

func CustomTest2(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus {
	r := scapiv1alpha3.TestResult{}
	r.Name = CustomTest2Name
	r.State = scapiv1alpha3.PassState
	r.Errors = make([]string, 0)
	r.Suggestions = make([]string, 0)
	almExamples := bundle.CSV.GetAnnotations()["alm-examples"]
	if almExamples == "" {
		fmt.Println("no alm-examples in the bundle CSV")
	}
	return wrapResult(r)
}

func wrapResult(r scapiv1alpha3.TestResult) scapiv1alpha3.TestStatus {
	return scapiv1alpha3.TestStatus{
		Results: []scapiv1alpha3.TestResult{r},
	}
}

## RESULTS:

[whayutin@thinkdoe oadp-operator-bundle]$ operator-sdk scorecard -x  bundle -o text  --selector=suite=foo -w "3000s" --kubeconfig /home/whayutin/.agnosticd/wdh0104pelorus1/ocp4-cluster_wdh0104pelorus1_kubeconfig    -s wes1 -n openshift-adp --verbose
DEBU[0000] Debug logging is set                         
--------------------------------------------------------------------------------
Image:      quay.io/rhn_engineering_whayutin/oadp_scorecard_oc:8
Entrypoint: [oadp-scorecard-test customtest2]
Labels:
	"suite":"foo"
	"test":"customtest2"
Results:
	Name: customtest2
	State: pass

	Log:
		namespace: openshift-adp


--------------------------------------------------------------------------------
Image:      quay.io/rhn_engineering_whayutin/oadp_scorecard_oc:8
Entrypoint: [oadp-scorecard-test customtest1]
Labels:
	"suite":"foo"
	"test":"customtest1"
Results:
	Name: customtest1
	State: pass

	Log:
		test:NAME                                                   READY   STATUS      RESTARTS   AGE
		pod/oadp-dpa-sample-1-aws-registry-f949cccd7-ksvjv     1/1     Running     0          2d4h
		pod/openshift-adp-controller-manager-8d5f75b49-gzz2d   1/1     Running     0          2d4h
		pod/restic-b4rcw                                       1/1     Running     0          2d4h
		pod/restic-m79sv                                       1/1     Running     0          2d4h
		pod/restic-qhwgh                                       1/1     Running     0          2d4h
		pod/scorecard-test-257w                                0/1     Completed   0          117m
		pod/scorecard-test-2kgf                                0/1     Completed   0          177m
		pod/scorecard-test-2sc4                                0/1     Error       0          6h7m
		pod/scorecard-test-4bj8                                0/1     Completed   0          5h37m
		pod/scorecard-test-4mbg                                0/1     Completed   0          97m
		pod/scorecard-test-6vxn                                0/1     Completed   0          179m
		pod/scorecard-test-7lg7                                0/1     Completed   0          5h13m
		pod/scorecard-test-7vkj                                0/1     Completed   0          164m
		pod/scorecard-test-8c9p                                0/1     Completed   0          4h56m
		pod/scorecard-test-8hlx                                0/1     Completed   0          163m
		pod/scorecard-test-8znf                                0/1     Error       0          6h6m
		pod/scorecard-test-94lc                                0/1     Error       0          136m
		pod/scorecard-test-99dm                                0/1     Completed   0          5h37m
		pod/scorecard-test-b2bf                                0/1     Completed   0          97m
		pod/scorecard-test-bkc5                                0/1     Completed   0          5h17m
		pod/scorecard-test-bqls                                0/1     Completed   0          5h7m
		pod/scorecard-test-bvsc                                0/1     Completed   0          5h6m
		pod/scorecard-test-bwkd                                0/1     Error       0          99m
		pod/scorecard-test-c82r                                0/1     Completed   0          91m
		pod/scorecard-test-cz5v                                0/1     Completed   0          91m
		pod/scorecard-test-fcbw                                0/1     Completed   0          136m
		pod/scorecard-test-fr82                                0/1     Completed   0          160m
		pod/scorecard-test-gd6p                                0/1     Completed   0          5h6m
		pod/scorecard-test-gx2f                                0/1     Completed   0          163m
		pod/scorecard-test-hgwh                                0/1     Error       0          5h17m
		pod/scorecard-test-hlhm                                0/1     Error       0          117m
		pod/scorecard-test-j9rn                                0/1     Completed   0          5h12m
		pod/scorecard-test-jl9z                                0/1     Completed   0          6h5m
		pod/scorecard-test-jp6v                                0/1     Completed   0          10m
		pod/scorecard-test-k8c2                                0/1     Completed   0          93m
		pod/scorecard-test-ksvz                                0/1     Completed   0          5h14m
		pod/scorecard-test-lknx                                0/1     Completed   0          164m
		pod/scorecard-test-llwm                                0/1     Error       0          160m
		pod/scorecard-test-m6c4                                0/1     Completed   0          174m
		pod/scorecard-test-mmnl                                0/1     Completed   0          179m
		pod/scorecard-test-nwxv                                0/1     Completed   0          5h6m
		pod/scorecard-test-qfn9                                0/1     Completed   0          5h7m
		pod/scorecard-test-r4cp                                0/1     Completed   0          4h56m
		pod/scorecard-test-s6rs                                0/1     Completed   0          5h12m
		pod/scorecard-test-sdxn                                1/1     Running     0          5m35s
		pod/scorecard-test-vj8g                                0/1     Error       0          99m
		pod/scorecard-test-vnzj                                0/1     Completed   0          10m
		pod/scorecard-test-wz2m                                0/1     Completed   0          5h6m
		pod/scorecard-test-xg27                                0/1     Completed   0          177m
		pod/scorecard-test-xs5n                                0/1     Completed   0          5m35s
		pod/scorecard-test-z2s4                                0/1     Completed   0          93m
		pod/scorecard-test-zbml                                0/1     Completed   0          6h5m
		pod/scorecard-test-zfpr                                0/1     Completed   0          174m
		pod/velero-99684c8c8-dbrvx                             1/1     Running     0          2d4h
		
		NAME                                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
		service/oadp-dpa-sample-1-aws-registry-svc                 ClusterIP   172.30.220.82   <none>        5000/TCP   2d4h
		service/openshift-adp-controller-manager-metrics-service   ClusterIP   172.30.174.1    <none>        8443/TCP   2d4h
		service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.176.27   <none>        8085/TCP   2d4h
		
		NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
		daemonset.apps/restic   3         3         3       3            3           <none>          2d4h
		
		NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
		deployment.apps/oadp-dpa-sample-1-aws-registry     1/1     1            1           2d4h
		deployment.apps/openshift-adp-controller-manager   1/1     1            1           2d4h
		deployment.apps/velero                             1/1     1            1           2d4h
		
		NAME                                                         DESIRED   CURRENT   READY   AGE
		replicaset.apps/oadp-dpa-sample-1-aws-registry-f949cccd7     1         1         1       2d4h
		replicaset.apps/openshift-adp-controller-manager-8d5f75b49   1         1         1       2d4h
		replicaset.apps/velero-99684c8c8                             1         1         1       2d4h
		
		NAME                                                            HOST/PORT                                                                                                         PATH   SERVICES                             PORT    TERMINATION   WILDCARD
		route.route.openshift.io/oadp-dpa-sample-1-aws-registry-route   oadp-dpa-sample-1-aws-registry-route-openshift-adp.apps.cluster-wdh0104pelorus1.wdh0104pelorus1.mg.dog8code.com          oadp-dpa-sample-1-aws-registry-svc   <all>                 None

## Prow workflow

** src could be here: https://github.com/redhat-openshift-ecosystem/operator-test-playbooks

I see the following in cvp logs

http://external-ci-coldstorage.datahub.redhat.com/cvp/cvp-redhat-operator-bundle-image-validation-test/oadp-operator-bundle-container-0.5.5-5/455b6083-9632-48f1-b4af-52ff105af38f/cvp-test-report.html

OpenShift CI deployment (bundle-image) -->
Run Operator Scorecard Tests (bundle-image) -->
Run Operator Custom Scorecard Tests (bundle-image)

http://external-ci-coldstorage.datahub.redhat.com/cvp/cvp-redhat-operator-bundle-image-validation-test/oadp-operator-bundle-container-0.5.5-5/455b6083-9632-48f1-b4af-52ff105af38f/operator-olm-deployment-bundle-image-output.txt -->

http://external-ci-coldstorage.datahub.redhat.com/cvp/cvp-redhat-operator-bundle-image-validation-test/oadp-operator-bundle-container-0.5.5-5/455b6083-9632-48f1-b4af-52ff105af38f/operator-scorecard-tests-bundle-image-output.txt -->

## Steps defined

https://steps.ci.openshift.org/workflow/optional-operators-cvp-common-claim

Krunoslav Pavic, 6 min, Edited
Hi, the CVP scorecard tests do execute after the operator is installed - even the default(basic) scorecard tests run with the assumption that the operator is deployed.
You can see the full Prow workflow and the sequence of individual steps here https://steps.ci.openshift.org/workflow/optional-operators-cvp-common-claim