# Release 2.33.0 - Manual test checklist

To best simulate the environment that our users will use our software
in, the manual tests should be done with the tools that are installed by
our installer.

Download the relevant tentative release from the GitLab [releases](https://gitlab.com/code-intelligence/core/-/releases)
page.

Alternatively, build and run the installer locally:

INSTALLER=$(./scripts/release/build-installer.sh) && \
./"${INSTALLER}" --non-interactive --install-vscode-extension

Then start the local server (with verbose logging to ease debugging):

ci-daemon -v2 --alsologtostderr

### Nightly Build
* [X] Check the CI/CD pipeline

### CI/CD scripts
* [ ] Test CI/CD scripts (Robert)
* [x] Start the `nightly` VM in Azure.
* [x] SSH into the `nightly` VM, download the installer from the tentative release and install it. You can find the VM's
SSH private key at https://gitlab.com/code-intelligence/core/-/settings/ci_cd with the name `NIGHTLY_VM_SSH_PRIVATE_KEY`.

    ssh -i <ssh private key>  azureuser@<vm ip>
    curl -o installer "<installer-URL>"
    chmod +x installer
    ./installer -n

* [ ] TODO(Rob): I tried the next step, but don't have access to the repository.
* [ ] Test the GitHub action
* At the [github-actions](https://github.com/CodeIntelligenceTesting/github-actions)
repository, merge the changes from the `v3` branch onto the `dev` branch.
* Update the images in the actions on the `dev` branch to use `<CICTL_VERSION>`
instead of `v3`, for example: `image: "docker://cifuzz/github-action:v3"`
will be updated to `image: "docker://cifuzz/github-action:<CICTL_VERSION>"`,
where `<CICTL_VERSION>` is the new cictl version.
* At nightly.code-intelligence.com, generate a GitHub Action
CI/CD script for the `zint` project on the `GitHub Projects` organization and update
[this file](https://github.com/CodeIntelligenceTesting/zint/blob/release-test/.github/workflows/main.yml).
* Replace
`${{ secrets.CI_FUZZ_API_TOKEN }}` with
`${{ secrets.CI_FUZZ_CICD_TEST_API_TOKEN }}`,
`branches: [ master ]` with `branches: [ release-test ]` in `push`,
`@v3` with `@dev`, and the port 443 with 6773 of the `FUZZING_SERVER_ADDRESS`.
* Check if the action will run and a finding is encountered.
* [ ] At nightly.code-intelligence.com, generate a GitLab CI/CD
script for the `zint-cicd-test` project and update
[this file](https://gitlab.com/ci-fuzz/zint-cicd-test/-/blob/release-test/.gitlab-ci.yml).
Replace `cifuzz/gitlab:v3` with `cifuzz/gitlab:<CICTL_VERSION>` where
`<CICTL_VERSION>` is the new cictl version and change the
`FUZZING_SERVER_URL` port 443 to 6773.
Check if the pipeline runs and a finding is encountered.
* [ ] At nightly.code-intelligence.com, generate a Jenkins
CI/CD script for the `zint` project on the `Jenkins Projects` organization.
Update [this Jenkinsfile](https://github.com/CodeIntelligenceTesting/zint/blob/release-test/Jenkinsfile)
with the recent script. Replace the `CICTL_URL` value with
`https://s3.eu-central-1.amazonaws.com/public.code-intelligence.com/cictl/cictl-<CICTL_VERSION>-linux`,
change the `FUZZING_SERVER_URL` port from 443 to 6773,
and `credentials('CI_FUZZ_API_TOKEN')` to
`credentials('CI_FUZZ_API_TOKEN_NIGHTLY')`. Check the output of the newest job at
https://jenkins.demo.code-intelligence.com/job/zint-nightly/job/release-test,
it should encounter a finding.
* [ ] Stop the `nightly` VM in Azure.

### Local
Manually test CI Fuzz with projects in the [fuzz-testing repository][1].

[1]: https://gitlab.com/code-intelligence/fuzz-testing

Before starting, make sure you set these variables in the `.env` file at the root of the core repository:

DOCKER_REGISTRY
DOCKER_REGISTRY_USER
DOCKER_REGISTRY_PASSWORD

If you don't have a `.env` file yet, copy `.env.template` to `.env` and replace the placeholders.

When you start to test a project, write your name in parenthesis behind the "Test $PROJECT" bullet item.

* [x] Test CppCMS (Norb)
* [x] Test creating and initializing the CppCMS project:
* Build Script:

      #!/bin/bash -eu
      mkdir build
      cd build
      cmake -DCMAKE_TESTING_ENABLED=OFF -DDISABLE_SHARED=ON -DDISABLE_GCRYPT=ON -DDISABLE_OPENSSL=ON -DDISABLE_FCGI=ON -DDISABLE_SCGI=ON -DDISABLE_HTTP=ON -DDISABLE_CACHE=ON -DDISABLE_TCPCACHE=ON -DDISABLE_GZIP=ON ..
      make -j$(nproc)

* Build Image: `cifuzz/builder-cppcms`
* [x] (VS Code) Test the fuzz-this-function functionality by trying to generate a fuzz target from an API function.
* [x] Start the "Fuzz Json" fuzz test by selecting it and clicking "Run".
* [x] Check that charts are showing and updating once the test collection is running.
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists. (You might need to stop the run for that.)
* [x] (VS Code) Insert a bug at line 1220 in the file "src/json.cpp"
(e.g., `*(char*)2=1;`). Re-run the campaign to produce a crash
finding. Debug the found crash.
* [x] (VS Code) Click on "All Findings" then click on the finding in the table. Make sure that clicking on the file link navigates you to the erroneous line.
* [x] (VS Code) Test if debugging the finding works and you can step to the error introduced
* [x] Test grammar fuzzer by clicking on the `fuzz_json_grammar` target and clicking "run" or "start run"
* [x] Test nginx (testing the socket fuzzer) (Norb)
* [x] Test creating and initializing a fuzzing project using
* The build script:

      #!/bin/bash -eu
      ./auto/configure
      mkdir -p logs
      make

* The build image: cifuzz/builder-nginx
* [x] Create a new socket fuzz test using the "add fuzz target"
functionality in the UI.
* [x] Select TCP port 6666 and Raw data.
* [x] Select the "Binary" `nginx` from the dropdown menu.
* [x] Use the following run arguments:

      -p
      $SRC
      -c
      .code-intelligence/nginx2.conf

* [x] Save it and add the following to the fuzz target yaml file at `.code-intelligence/fuzz_targets`:

      engine_options:
        libfuzzer:
          - "-detect_leaks=0"

* [x] Run the the created fuzz test by selecting it and clicking "run" or "start run".
* [x] Stop the run.
* [x] Test struct fuzzer (core repository) (Norb)
* [x] Run the struct fuzzer (called "do_struct_fuzzer") in the example project
in `core/fuzzing/testdata/projects/example/` using (A crash should be found quickly):
* Build Script: `make`
* Build Image: cifuzz/builder-cppcms
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists.
* [x] (VS Code) Debug the crash and verify that you can step through the code
and see the value of the struct that caused the crash.

* [x] Test string instrumentation in the example java project (Norb)
* [x] Create and initialize the project (found in the core repo:
fuzzing/testdata/projects/java_example) with the settings:
* build script:

       mkdir -p build/libs
       javac $(find src -name "*.java") -d build
       jar cf build/libs/example.jar -C build com

* image: cifuzz/builder-maven
* [X] Select the fuzz test "FuzzParser" that is already configured and run it.
* [x] (VS Code) Test debugging the finding, and step into the code until you reach the bug location.
* [x] Test WebGoat (Robert)
* [x] Build WebGoat manually using maven `mvn clean package -DskipTests`
* [ ] TODO(rob): Format
* [x] Initialize a fuzzing project for WebGoat
* [x] Select the "Configure project for web application fuzzing" option to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "org.owasp" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Run

      java -jar <javaagent_command> webgoat-server/target/webgoat-server-8.0.0-SNAPSHOT.jar

* [x] WebGoat should start and appear in the list of web services
* [x] Add a new "Web App Fuzz Test"
* [x] Leave all default options
* [ ] TODO(rob): Select your Webservice
* [x] In a browser navigate to `http://localhost:8080/WebGoat` and create a user with username `testuser` and password `testpass`
* [ ] TODO(rob): unindent
* [x] In the file `.code-intelligence/fuzz_targets/<fuzz-target-name>_initial_request.http` insert the following login request

      POST /WebGoat/login HTTP/1.1
      Host: localhost:8080
      Content-Type: application/x-www-form-urlencoded
      Content-Length: 35
      Origin: http://localhost:8080
      Referer: http://localhost:8080/WebGoat/login

      username=testuser&password=testpass

* [x] Start the created Fuzz Test by selecting it and clicking "Run".
* [x] A "Possible SQL Injection" finding and multiple exceptions should be reported
* [x] Verify that the cURL command for the SQL Injection finding reproduces the SQL Exception
* [x] Verify that there are multiple warnings reported from the ZAP run (might take up to 3 minutes).
* Note: ZAP run findings have a locations like `POST http://127.0.0.1:8080/[…]` or `GET http://127.0.0.1:8080/[…]`, examples include types like
* "Application Error Disclosure",
* "Information Disclosure […]",
* "Cookie without SameSite Attribute",
* "Cookie No HttpOnly Flag" or
* "Charset Mismatch […]".
* [x] Click on the "Overview" page and verify that coverage is reported
* [x] Click on "Code Coverage" and check that the coverage view works
* [ ] TODO(rob): uncheck
* [x] Click on some file that has 0%<coverage<100% and check if source code opens and some lines are highlighted and they make sense

* [x] Test AltoroJ (Norb)
* [x] Run AltoroJ locally with Tomcat
* [x] If not installed yet download extract Tomcat 7 or 8 (https://tomcat.apache.org/download-80.cgi)
* [x] Download the AltoroJ application from https://github.com/HCL-TECH-SOFTWARE/AltoroJ
* [x] Build the AltoroJ application with `gradle build` using Gradle version 6.x
* [x] Copy the web application `build/libs/altoromutual.war` to `<Tomcat_install_dir>/web_apps/AltoroJ.war`
* [x] Start the tomcat server with `<Tomcat_install_dir>/bin/startup.sh`
* [x] The AltoroJ application should be available on `http://localhost:8080/AltoroJ`
* [x] Initialize the fuzzing project AltoroJ
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "com.ibm" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Stop the running Tomcat server (`<Tomcat_install_dir>/bin/shutdown.sh`)
* [x] Start it again with the copied java agent argument `JAVA_OPTS=<copied_command> <Tomcat_install_dir>/bin/startup.sh`
* [x] The web service should appear in the list of web services
* [x] In the `.code-intelligence/web_services.yaml` configure the OpenAPI
spec for the added web service: `WebContent/swagger/properties.json`.
* [x] Create a new "Java Web App Fuzz Test" and select the web service configured before
* [x] Create a file `.code-intelligence/fuzz_targets/<name_of_fuzz_test>_headers.sh` and insert the following content:

    token=$(curl -s -X POST "http://localhost:8080/AltoroJ/api/login" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{  \"username\": \"jsmith\",  \"password\": \"demo1234\"}" | jq .Authorization | tr -d '"')
    echo "Authorization: $token"

You can execute the script to verify that the Authorization token is successfully extracted
* [x] Run the create fuzz test and verify that an SQL
injection and more then 10 exceptions are found.
* [x] Verify that ZAP findings are reported (might take ~2 minutes)
* [x] Verify that code coverage is reported
* [x] Test 32-bit fuzzing (Norb)
* [x] Initialize the project `32-bit_fuzzing` from the fuzz-testing repo
* Build script is: ```clang software_under_test.c -c -m32 -o software_under_test.o```
* Docker image is: `cifuzz/builder-c:1.1`
* [x] Execute the fuzz test `32-bit_fuzz_test` and check if a double free is found
* [x] Test file input fuzzing (Norb)
* [x] Initialize the project `file_input_fuzzing` from the fuzz-testing repo
* Build script can be empty
* [x] Execute the fuzz test `file_input_fuzzing_64-bit` and check if a stack buffer overflow is found
* [x] Test Cmake Integration (Norb)
* [x] Open the `cmake_integration` folder from the fuzz-testing repo and adjust the `CIFUZZ_INSTALL_ROOT` in `build_fuzzing_artifacts_and_run_fuzz_test.sh`
* [x] Run the `build_fuzzing_artifacts_and_run_fuzz_test.sh` script
* [x] Confirm that the script was run with the correct version of CI Fuzz, that the fuzz test was build successfully and that the crashing input starts with "CI FUZZ"
* [x] Test the unit test runner
* [ ] TODO(rob): Remove this test
* [x] Test java-grpc fuzzing with greeter service (Norb)
* [x] In the core repo, build the greeter service by running

    ./bazelisk.py build //pkg/web_app/tests/integration/grpc/testdata/java/greeter_service:GreeterServer_deploy.jar

* [x] In the core repo, run

    ci-protoc libproto_stub.so  -Ipkg/web_app/tests/integration/grpc/testdata/common/greeter_service  pkg/web_app/tests/integration/grpc/testdata/common/greeter_service/helloworld/a/b/c/d/greeter_service.proto  pkg/web_app/tests/integration/grpc/testdata/common/greeter_service/helloworld/a/b/c2/d2/greeter_messages.proto   --field_hint=authentication.access_token=let_me_in  --field_hint=authentication.access_token=backdoor

to build the libproto_stub.so.
* [x] Move libproto_stub.so the to greeter service project dir

    mv libproto_stub.so pkg/web_app/tests/integration/grpc/testdata/java/greeter_service

* [x] Open the greeter service directory `pkg/web_app/tests/integration/grpc/testdata/java/greeter_service` in VS Code and initialize a web application fuzzing project
* [x] Add a new webservice
* [x] Chose a name (e.g. `test`) and instrument `com.example`
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Start the previously build jar with the java agent command and specify the port

    java <javaagent_command> -Dserver.port=12345 -jar bazel-bin/pkg/web_app/tests/integration/grpc/testdata/java/greeter_service/GreeterServer_deploy.jar

* [x] Add a new "Web App Fuzz Test"
* [x] Select the previously created webservice
* [x] Modify the fuzz test config `.code-intelligence/fuzz_targets/<fuzztestname>.yaml` and replace the content with (change name of webservice from `test` if necessary)

    type: "java web app"
    run_extra_args:
      - "--proto_stub_path=libproto_stub.so"
    web_services:
      - "test"
    base_url: "127.0.0.1:12345"
    protocol: grpc

* [x] Create the file `.code-intelligence/fuzz_targets/<fuzztestname>_headers.http` with the content

    Authorization: Bearer letmein

* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown
* [x] Verify that you get 3 findings: Two are a `Security Exception`, one with `access_token: "backdoor"`, and one is a `Dereference of Null Object` with `access_token: "let_me_in"` in the Human-readable input.
* [x] Verify that the grpcurl reproducer commands work by running them. The output should be

    ERROR:
      Code: Unknown
      Message:

* [x] Stop the run (if not already done)
* [x] Go to "Code Coverage" and verify the coverage metrics are shown.
**TODO: Shown code coverage in files is off**

* [x] Test tidb (currently branch go-in-process) (Robert)
* [x] Checkout branch `go-in-process` in "fuzz-testing":

    git checkout go in process

* [ ] TODO(rob): Add hyphens
* [x] Execute

    go mod tidy

* [x] Test creating and initializing a fuzzing project using
* [x] The build script:

      exit 0

* [x] Build in local mode
* [x] Under "Additional Docker volume mounts", add mount to GO_PATH (usually `$HOME/go` to `$HOME/go`)
* Note if your local go installation lives somewhere else, change "Source" accordingly.
* "Destination" still needs to be `$HOME/go`.
* [x] Enable the hidden Fuzz Targets:
* [x] Fuzz Target creation in VSCODE UI is not supported and not planned. You need to add project.yaml entry:

      go_fuzz_packages:
      - github.com/pingcap/tidb/types

* [x] In the VSCode UI you should see afterwards that three fuzztargets were found in the specified package.
* [x] Run the FuzzMarshalJSON target and verify that fuzzing metrics graphs are shown
* [x] Run the FuzzNewHexLiteral target and verify that we get a "Go Panic" finding
* [ ] Verify that you can jump to the bug location from the UI
* [x] Verify that the stack trace contains three elements and that you can jump to the corresponding location
* [x] Add a new fuzz test to the file types/fuzzNewHexLiteral.go (copy/paste/rename the existing fuzz test FuzzNewHexLiteral). Verify that the newly added fuzz test appears in the list of fuzz targets and that you can go to the file from the UI. (This takes several seconds, have patience!)

* [x] Test `go-grpc` fuzzing with greeter service (Norb)
* [x] In the [grpc-go](https://gitlab.com/code-intelligence/fuzz-testing/-/tree/master/grpc-go)
project from the fuzz-testing repo build `libproto_stub.so` via:

    cd examples
    ci-protoc libproto_stub.so -Ihelloworld/helloworld/ helloworld/helloworld/helloworld.proto

* [x] Create and initialize a fuzzing project in VS Code
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name e.g. go-grpc-server for the service
* [x] Click on "go" and follow the instructions in the terminal. They should be similar to:

      ci-gofuzz --include "google.golang.org/grpc/examples/helloworld*" \
      -cover \
      -cifuzz_agent_lib $HOME/bin/libfuzzing_agent.so \
      -o output_binary_with_instrumentation ./helloworld/greeter_server/

* [x] Start the SUT, but replace the CIFUZZ_SERVICE_NAME with your generated service name in the instructions:

      LD_LIBRARY_PATH=$HOME/bin/ GLOG_v=2 GLOG_alsologtostderr=1 \
      CIFUZZ_SERVICE_NAME=projects/grpc-go-bf1747a2/web_services/go-grpc-helloword2 \
      ./output_binary_with_instrumentation

* [x] Add a Fuzz test with name `test` and modify the fuzz test config `.code-intelligence/fuzz_targets/test.yaml` and replace the content with

    type: "java web app"
    run_extra_args:
      - "--proto_stub_path=examples/libproto_stub.so"
    web_services:
      - "go-grpc-server"
    base_url: "127.0.0.1:50051"
    protocol: grpc
    tag_header: X-Request-Id

* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown. If the crash occurs too fast it can happen that no metrics are shown
* [x] Verify that you get 2 findings:
* [x] Verify that the fuzz target stops.
* [x] One is a `Go Panic` with `name: "Critical Bug"` and the second a `Crash (exit code: 2)` also with `name: "Critical Bug"` in the Human-readable input. Check that the first one has reasonable SUMMARY, DETAILS, STACKTRACE and LOG in the finding. The second finding is expected to only have Content in SUMMARY and DETAILS.
* [x] Check that the summary contains a request tag header called `X-Request-Id` as specified in the fuzz test config.
* [x] Verify that you can jump to the bug location from the UI
* [x] Verify that one of the grpcurl reproducer commands work by running them. It will not work, because the SUT does not support the reflection API. You will need to add the proto file path to the grpcurl command ´-proto helloworld/helloworld/helloworld.proto -import-path helloworld/helloworld´. The command should look similar to

    grpcurl \
    -proto helloworld/helloworld/helloworld.proto -import-path helloworld/helloworld \
    -plaintext \
    -d '{"name":"Critical Bug"}' \
    127.0.0.1:50051 helloworld.Greeter/SayHello

* [x] Uncomment the vulnerability (lines 45-47) in grpc-go/examples/helloworld/main.go and recompile the fuzztarget again.

    ci-gofuzz --include "google.golang.org/grpc/examples/helloworld*" \
    -cover \
    -cifuzz_agent_lib $HOME/bin/libfuzzing_agent.so \
    -o output_binary_with_instrumentation ./helloworld/greeter_server/

* [x] Restart application again with correct service name:

    LD_LIBRARY_PATH=$HOME/bin/ GLOG_v=2 GLOG_alsologtostderr=1 \
    CIFUZZ_SERVICE_NAME=projects/grpc-go-bf1747a2/web_services/go-grpc-helloword2 \
    ./output_binary_with_instrumentation

* [x] Set fuzz target runtime to 5 minutes in project.yaml
* [x] Set coverage intervall to 10000 by adding `coverage_update_interval: 10000` to the fuzz target yaml.
* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown.
* [x] Verfiy that the Source Code Coverage is available during runtime after a minute
* [x] Stop the fuzz target. Let the target application run withouth termination.
* [x] Check that the webservice light goes back to green again
* [x] Restart fuzzing and check that fuzzing is successfully started again.